Libraries tagged by crawling

This is a Wrapper for the API provided by most Wordpress Blogs. Intended for easy crawling of a Wordpress page.

Go to Download

Sigmie Crawler is a PHP package for crawling websites and exporting their HTML contents.

Go to Download

Ant: For 4chan image crawling

Go to Download

Webcrawler for crawling indexed sites i.e. options and indexes

Go to Download

A generic rate limiter for the web and reactphp. Useful for API clients, web crawling, or other tasks that need to be throttled

Go to Download

a library for crawling search engine results and extracting metadata for a set of keywords

Go to Download

Rake 2.0 core crawling/migration library

Go to Download

A powerful WordPress plugin for data migration and web crawling using Rake 2.0 framework

Go to Download

Universal package for BrightData web scraping and crawling. And making keyword research easier.

Go to Download

Crawling rules processor

Go to Download

Analyzing and crawling the meta tags of a static website

Go to Download

Analyzing and crawling the html structure of a static/dynamic website

Go to Download

cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot.

Go to Download

Toolkit that helps in creating custom APIs using cURL HTTP requests and crawling thru the DOM

Go to Download

A quick-start helper to website crawling.

Go to Download