Libraries tagged by crawling
duzun/hquery
90187 Downloads
An extremely fast web scraper that parses megabytes of HTML in a blink of an eye. No dependencies. PHP5+
crwlr/crawler
2659 Downloads
Web crawling and scraping library.
stil/curl-easy
188258 Downloads
cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot.
cyber-duck/silverstripe-seo
47177 Downloads
A SilverStripe module to optimise the Meta, crawling, indexing, and sharing of your website content
proxycrawl/proxycrawl
60038 Downloads
A lightweight, dependency free PHP class that acts as wrapper for ProxyCrawl API
crwlr/robots-txt
5113 Downloads
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
cleantalk/anti-ddos-lite
741 Downloads
A small PHP app to protect your site against DDoS attack or crawling web site by bots.
fourlabs/robots-bundle
19006 Downloads
Symfony2 bundle to control X-Robots-Tag HTTP header via annotations
crawlbase/crawlbase
561 Downloads
A lightweight, dependency free PHP class that acts as wrapper for Crawlbase API
renoki-co/clusteer
1172 Downloads
Clusteer is a Puppeteer wrapper written for PHP, with the super-power of parallelizing pages across multiple browser instances.
plasticstudio/silverstripe-seo
4109 Downloads
A SilverStripe module to optimise the Meta, crawling, indexing, and sharing of your website content (forked from Cyber-Duck/Silverstripe-SEO)
laurentvw/scrapher
2510 Downloads
A web scraper for PHP to easily extract data from web pages
dexiio/dexi-api-client
9743 Downloads
Dexi API Client for PHP 5.3+
crwlr/crawler-ext-browser
130 Downloads
Extension for the crwlr/crawler package containing steps utilizing a headless browser.
crawlzone/crawlzone
5004 Downloads
Crawlzone is a fast asynchronous internet crawling framework aiming to provide open source web search and testing solution. It can be used for a wide range of purposes, from extracting and indexing structured data to monitoring and automated testing.