Libraries tagged by web crawler
jaybizzle/laravel-crawler-detect
2211941 Downloads
A Laravel package to detect web crawlers via the user agent
jaeger/querylist
618461 Downloads
Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫)，基于phpQuery。
vdb/php-spider
173080 Downloads
A configurable and extensible PHP web spider
spekulatius/phpscraper
122258 Downloads
PHPScraper, built with simplicity in mind. See tests/ for more examples.
terminal42/escargot
1203950 Downloads
A web crawler or spider library based on Symfony components
crwlr/crawler
10474 Downloads
Web crawling and scraping library.
vipnytt/robotstxtparser
704356 Downloads
Robots.txt parsing library, with full support for every directive and specification.
vipnytt/useragentparser
882355 Downloads
User-Agent parser for robot rule sets
opensearchserver/opensearchserver
65664 Downloads
PHP library for OpenSearchServer: professionnal search engine, crawlers (web, file, database), REST APIs, .... This library uses OpenSearchServer's V2 API.
blackfire/player
16303 Downloads
A powerful web crawler and web scraper with Blackfire support
eddieace/php-simple
55205 Downloads
crwlr/robots-txt
17496 Downloads
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
cuab/phpcrawl
36837 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
katsana/dusk-crawler
12482 Downloads
Web Crawler using Laravel Dusk
crawlzone/crawlzone
5262 Downloads
Crawlzone is a fast asynchronous internet crawling framework aiming to provide open source web search and testing solution. It can be used for a wide range of purposes, from extracting and indexing structured data to monitoring and automated testing.