Libraries tagged by crawl
crwlr/crawler
10163 Downloads
Web crawling and scraping library.
nmure/crawler-detect-bundle
271555 Downloads
A Symfony bundle for the Crawler-Detect library (detects bots/crawlers/spiders via the user agent)
crawlbase/crawlbase
32225 Downloads
A lightweight, dependency free PHP class that acts as wrapper for Crawlbase API
vipnytt/robotstxtparser
694805 Downloads
Robots.txt parsing library, with full support for every directive and specification.
spatie/http-status-check
47805 Downloads
CLI tool to crawl a website and check HTTP status code
proxycrawl/proxycrawl
85981 Downloads
A lightweight, dependency free PHP class that acts as wrapper for ProxyCrawl API
jyggen/curl
146134 Downloads
A simple and lightweight cURL library with support for asynchronous requests.
stil/curl-easy
201314 Downloads
cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot.
baba/sitemap-crawler
Downloads
smochin/instagram-php-crawler
8374 Downloads
A simple PHP Crawler for Instagram
friends-of-hyva/magento2-crawler-session
3678 Downloads
Prevent crawlers from creating a session
dachcom-digital/dynamic-search-data-provider-crawler
25605 Downloads
aoepeople/crawler
287765 Downloads
Crawler extension for TYPO3
luka-dev/headless-task-server-php
8851 Downloads
Helper for sending requests to luka-dev/headless-task-server
vipnytt/useragentparser
871517 Downloads
User-Agent parser for robot rule sets