Libraries tagged by crawling

An extremely fast web scraper that parses megabytes of HTML in a blink of an eye. No dependencies. PHP5+

Go to Download

Web crawling and scraping library.

Go to Download

cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot.

Go to Download

PHP SDK for the Firecrawl v2 API with Laravel support

Go to Download

A lightweight, dependency free PHP class that acts as wrapper for Crawlbase API

Go to Download

A lightweight, dependency free PHP class that acts as wrapper for ProxyCrawl API

Go to Download

A small PHP app to protect your site against DDoS attack or crawling web site by bots.

Go to Download

Generates an llms.txt file according to the llmstxt.org specification and serves pages as Markdown, giving Large Language Models a structured, machine-readable view of your site.

Go to Download

A SilverStripe module to optimise the Meta, crawling, indexing, and sharing of your website content

Go to Download

PHP SDK for Firecrawl API - Web scraping and crawling service

Go to Download

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Go to Download

Framework-agnostic IndexNow client and queue for submitting URL changes to participating search engines (Bing, Yandex, Seznam, Naver, Yep).

Go to Download

Clusteer is a Puppeteer wrapper written for PHP, with the super-power of parallelizing pages across multiple browser instances.

Go to Download

A SilverStripe module to optimise the Meta, crawling, indexing, and sharing of your website content (forked from Cyber-Duck/Silverstripe-SEO)

Go to Download

Crawlzone is a fast asynchronous internet crawling framework aiming to provide open source web search and testing solution. It can be used for a wide range of purposes, from extracting and indexing structured data to monitoring and automated testing.

Go to Download