Libraries tagged by crawler

A powerful web crawler and web scraper with Blackfire support

Go to Download

A lightweight, dependency free PHP class that acts as wrapper for ProxyCrawl API

Go to Download

A Symfony bundle for the Crawler-Detect library (detects bots/crawlers/spiders via the user agent)

Go to Download

A highly extendible, dependency free Crawler for HTML, PDFS or any other type of Documents.

Go to Download

Prevent crawlers from creating a session

Go to Download

Site Crawler - TYPO3 Crawler crawls the TYPO3 page tree. Used for cache warmup, indexing, publishing applications etc.

Go to Download

Go to Download

User-Agent parser for robot rule sets

Go to Download

Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page

Go to Download

Package to provide api-like access to foreign sites based on html parsing

Go to Download

Find dead links, 404, 301 and more (command line or admin).

Go to Download

PHP library for OpenSearchServer: professionnal search engine, crawlers (web, file, database), REST APIs, .... This library uses OpenSearchServer's V2 API.

Go to Download

Last.fm webservice client for php.

Go to Download

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性

Go to Download

PHP SDK for Firecrawl API - Web scraping and crawling service

Go to Download