Libraries tagged by crawling

An extremely fast web scraper that parses megabytes of HTML in a blink of an eye. No dependencies. PHP5+

Go to Download

cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot.

Go to Download

Web crawling and scraping library.

Go to Download

A lightweight, dependency free PHP class that acts as wrapper for Crawlbase API

Go to Download

A lightweight, dependency free PHP class that acts as wrapper for ProxyCrawl API

Go to Download

A SilverStripe module to optimise the Meta, crawling, indexing, and sharing of your website content

Go to Download

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Go to Download

A small PHP app to protect your site against DDoS attack or crawling web site by bots.

Go to Download

Crawlzone is a fast asynchronous internet crawling framework aiming to provide open source web search and testing solution. It can be used for a wide range of purposes, from extracting and indexing structured data to monitoring and automated testing.

Go to Download

A Magento 2 module that improves store performance by automatically warming cache through intelligent page crawling and async refresh mechanisms.

Go to Download

Clusteer is a Puppeteer wrapper written for PHP, with the super-power of parallelizing pages across multiple browser instances.

Go to Download

A SilverStripe module to optimise the Meta, crawling, indexing, and sharing of your website content (forked from Cyber-Duck/Silverstripe-SEO)

Go to Download

Dexi API Client for PHP 5.3+

Go to Download

Translations Manager extension to use Google translation crawling to auto translate your __(), trans() fn

Go to Download

Parallel URL crawling extension to curl-easy.

Go to Download