Libraries tagged by web crawler
jaybizzle/laravel-crawler-detect
2722342 Downloads
A Laravel package to detect web crawlers via the user agent
jaeger/querylist
656449 Downloads
Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫),基于phpQuery。
crwlr/crawler
17197 Downloads
Web crawling and scraping library.
vipnytt/robotstxtparser
786388 Downloads
Robots.txt parsing library, with full support for every directive and specification.
vdb/php-spider
185507 Downloads
A configurable and extensible PHP web spider
spekulatius/phpscraper
153365 Downloads
PHPScraper, built with simplicity in mind. See tests/ for more examples.
terminal42/escargot
1481734 Downloads
A web crawler or spider library based on Symfony components
crwlr/robots-txt
34145 Downloads
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
vipnytt/useragentparser
982441 Downloads
User-Agent parser for robot rule sets
opensearchserver/opensearchserver
67983 Downloads
PHP library for OpenSearchServer: professionnal search engine, crawlers (web, file, database), REST APIs, .... This library uses OpenSearchServer's V2 API.
blackfire/player
17076 Downloads
A powerful web crawler and web scraper with Blackfire support
revolution/salvager
9667 Downloads
Tiny WebCrawler for Laravel using Playwright
eddieace/php-simple
64403 Downloads
cuab/phpcrawl
39899 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
katsana/dusk-crawler
13032 Downloads
Web Crawler using Laravel Dusk