Libraries tagged by crawling

A generic rate limiter for the web and reactphp. Useful for API clients, web crawling, or other tasks that need to be throttled

Go to Download

Lightweight library to crawl all the urls or links in a paragraph and replace with html anchor tags. Crawling is perfect.

Go to Download

A simple spider crawling web pages from baike.baidu.com

Go to Download

Control Google's crawling and indexing of your Magento store with precision. SEO Indexation Rules applies noindex, nofollow, and related robots directives to parameterised URLs, full action names, and URL path patterns — including automatic noindex for staging hostnames.

Go to Download

Detect Seznam.cz bot crawling websites. Detection for page crawler and thumbnail screening bot.

Go to Download

Most powerful, popular and production crawling/scraping package for PHP

Go to Download

cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot.

Go to Download

Browser emulator, for crawling web pages

Go to Download

Sigmie Crawler is a PHP package for crawling websites and exporting their HTML contents.

Go to Download

Ant: For 4chan image crawling

Go to Download

Webcrawler for crawling indexed sites i.e. options and indexes

Go to Download

A generic rate limiter for the web and reactphp. Useful for API clients, web crawling, or other tasks that need to be throttled

Go to Download

a library for crawling search engine results and extracting metadata for a set of keywords

Go to Download

Rake 2.0 core crawling/migration library

Go to Download

A powerful WordPress plugin for data migration and web crawling using Rake 2.0 framework

Go to Download