Libraries tagged by crawler
crwlr/robots-txt
37640 Downloads
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
spatie/laravel-link-checker
53388 Downloads
Check all links in a Laravel app
jyggen/curl
150604 Downloads
A simple and lightweight cURL library with support for asynchronous requests.
zrashwani/arachnid
20413 Downloads
A crawler to find all unique internal pages on a given website
zenstruck/dom
12142 Downloads
DOM crawler with advanced selector API and assertions.
schliesser/sitecrawler
38420 Downloads
TYPO3 sitemap crawler
vlados/laravel-blade-crawler-detect
5571 Downloads
Boost Lighthouse and PageSpeed scores by hiding cookie banners, chat widgets and heavy scripts from crawlers via two Blade directives. Octane-safe.
shel/crawler
2972 Downloads
Allows crawling of sitemaps and node-trees
migliori/sitemap-crawler
3960 Downloads
Sitemap crawler/generator. For the given URL it will return sitemap XML file with URLs and images.
luyadev/luya-module-crawler
33984 Downloads
An full search page crawler to enable complex and customized searching abilities.
dachcom-digital/dynamic-search-data-provider-crawler
31647 Downloads
alikdex/yii2-crawler-detect
22787 Downloads
Yii2 extension for CrawlerDetect library
elboletaire/crawler
6728 Downloads
A simple crawler (spider) writen in php just for fun
webignition/symfony-panther-dom-crawler-navigator
21581 Downloads
Navigate around a symfony panther dom crawler using convenience methods
katsana/dusk-crawler
13188 Downloads
Web Crawler using Laravel Dusk