Libraries tagged by crawling

An extremely fast web scraper that parses megabytes of HTML in a blink of an eye. No dependencies. PHP5+

Go to Download

cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot.

Go to Download

Web crawling and scraping library.

Go to Download

A lightweight, dependency free PHP class that acts as wrapper for Crawlbase API

Go to Download

TYPO3 extension for generating .well-known/llm.txt links according to llmstxt.org specification

Go to Download

PHP SDK for the Firecrawl v2 API with Laravel support

Go to Download

A lightweight, dependency free PHP class that acts as wrapper for ProxyCrawl API

Go to Download

PHP SDK for Firecrawl API - Web scraping and crawling service

Go to Download

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Go to Download

A small PHP app to protect your site against DDoS attack or crawling web site by bots.

Go to Download

Framework-agnostic IndexNow client and queue for submitting URL changes to participating search engines (Bing, Yandex, Seznam, Naver, Yep).

Go to Download

A SilverStripe module to optimise the Meta, crawling, indexing, and sharing of your website content (forked from Cyber-Duck/Silverstripe-SEO)

Go to Download

A SilverStripe module to optimise the Meta, crawling, indexing, and sharing of your website content

Go to Download

Crawlzone is a fast asynchronous internet crawling framework aiming to provide open source web search and testing solution. It can be used for a wide range of purposes, from extracting and indexing structured data to monitoring and automated testing.

Go to Download

Translations Manager extension to use Google translation crawling to auto translate your __(), trans() fn

Go to Download