Libraries tagged by crawl
stil/curl-easy
202946 Downloads
cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot.
crwlr/crawler
10727 Downloads
Web crawling and scraping library.
crawlbase/crawlbase
37011 Downloads
A lightweight, dependency free PHP class that acts as wrapper for Crawlbase API
vipnytt/robotstxtparser
707574 Downloads
Robots.txt parsing library, with full support for every directive and specification.
proxycrawl/proxycrawl
89071 Downloads
A lightweight, dependency free PHP class that acts as wrapper for ProxyCrawl API
baba/sitemap-crawler
Downloads
smochin/instagram-php-crawler
8752 Downloads
A simple PHP Crawler for Instagram
nmure/crawler-detect-bundle
272947 Downloads
A Symfony bundle for the Crawler-Detect library (detects bots/crawlers/spiders via the user agent)
friends-of-hyva/magento2-crawler-session
4179 Downloads
Prevent crawlers from creating a session
dachcom-digital/dynamic-search-data-provider-crawler
26901 Downloads
aoepeople/crawler
288125 Downloads
Crawler extension for TYPO3
gsouf/chromium
520 Downloads
Instrument headless chrome/chromium instances from PHP
vipnytt/useragentparser
886567 Downloads
User-Agent parser for robot rule sets
tomverran/robots-txt-checker
55130 Downloads
Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page
spatie/http-status-check
47888 Downloads
CLI tool to crawl a website and check HTTP status code