Libraries tagged by CrawlLib

A generic rate limiter for the web and reactphp. Useful for API clients, web crawling, or other tasks that need to be throttled

Go to Download

Lightweight library to crawl all the urls or links in a paragraph and replace with html anchor tags. Crawling is perfect.

Go to Download

A simple spider crawling web pages from baike.baidu.com

Go to Download

Detect Seznam.cz bot crawling websites. Detection for page crawler and thumbnail screening bot.

Go to Download

Most powerful, popular and production crawling/scraping package for PHP

Go to Download

Class to help with crawling web pages

Go to Download

cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot.

Go to Download

Browser emulator, for crawling web pages

Go to Download

This is a Wrapper for the API provided by most Wordpress Blogs. Intended for easy crawling of a Wordpress page.

Go to Download

Sigmie Crawler is a PHP package for crawling websites and exporting their HTML contents.

Go to Download

Ant: For 4chan image crawling

Go to Download

Webcrawler for crawling indexed sites i.e. options and indexes

Go to Download

A generic rate limiter for the web and reactphp. Useful for API clients, web crawling, or other tasks that need to be throttled

Go to Download

a library for crawling search engine results and extracting metadata for a set of keywords

Go to Download

Rake 2.0 core crawling/migration library

Go to Download