Libraries tagged by crawl
vipnytt/useragentparser
1017256 Downloads
User-Agent parser for robot rule sets
tomverran/robots-txt-checker
66455 Downloads
Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page
opensearchserver/opensearchserver
68292 Downloads
PHP library for OpenSearchServer: professionnal search engine, crawlers (web, file, database), REST APIs, .... This library uses OpenSearchServer's V2 API.
helgesverre/firecrawl-php-sdk
14598 Downloads
PHP SDK for Firecrawl API - Web scraping and crawling service
fkrzski/robots-txt
2680 Downloads
A modern, fluent PHP package for managing robots.txt rules with type safety and great developer experience
firecrawl/firecrawl-sdk
5778 Downloads
PHP SDK for the Firecrawl v2 API with Laravel support
eddieace/php-simple
65827 Downloads
crwlr/robots-txt
37584 Downloads
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
spatie/laravel-link-checker
53388 Downloads
Check all links in a Laravel app
jyggen/curl
150604 Downloads
A simple and lightweight cURL library with support for asynchronous requests.
zenstruck/dom
12134 Downloads
DOM crawler with advanced selector API and assertions.
schliesser/sitecrawler
38417 Downloads
TYPO3 sitemap crawler
cleantalk/anti-ddos-lite
4882 Downloads
A small PHP app to protect your site against DDoS attack or crawling web site by bots.
webignition/symfony-panther-dom-crawler-navigator
21581 Downloads
Navigate around a symfony panther dom crawler using convenience methods
shel/crawler
2972 Downloads
Allows crawling of sitemaps and node-trees