Libraries tagged by CrawlLib

Microservice for the password strength checker 'correct-horse-battery-staple'

Go to Download

A password strength checker

Go to Download

An easy-to-use PHP class for scraping webpages' source code.

Go to Download

Symfony2 bundle to control X-Robots-Tag HTTP header via annotations

Go to Download

Cquery PHP Scraping with Panther Client Loader

Go to Download

Crawles web pages

Go to Download

A web scraper for PHP to easily extract data from web pages -> party of laurentvw

Go to Download

A simple, flexible crawler, written in PHP.

Go to Download

Vue.js library for Yii2 based on prawee

Go to Download

cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot.

Go to Download

A generic rate limiter for the web and reactphp. Useful for API clients, web crawling, or other tasks that need to be throttled

Go to Download

Lightweight library to crawl all the urls or links in a paragraph and replace with html anchor tags. Crawling is perfect.

Go to Download

A simple spider crawling web pages from baike.baidu.com

Go to Download

Control Google's crawling and indexing of your Magento store with precision. SEO Indexation Rules applies noindex, nofollow, and related robots directives to parameterised URLs, full action names, and URL path patterns — including automatic noindex for staging hostnames.

Go to Download

Detect Seznam.cz bot crawling websites. Detection for page crawler and thumbnail screening bot.

Go to Download