Libraries tagged by spiders

CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

Go to Download

A Laravel package to detect web crawlers via the user agent

Go to Download

Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫)，基于phpQuery。

Go to Download

A configurable and extensible PHP web spider

Go to Download

A web crawler or spider library based on Symfony components

Go to Download

Robots.txt parsing library, with full support for every directive and specification.

Go to Download

PHP Wrapper for Curl

Go to Download

wrap curl client (http client) for PHP 5.3; using php multi curl, parallel request and write asynchronous code

Go to Download

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Go to Download

A Symfony bundle for the Crawler-Detect library (detects bots/crawlers/spiders via the user agent)

Go to Download

This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.

Go to Download

A crawler to find all unique internal pages on a given website

Go to Download

User-Agent parser for robot rule sets

Go to Download

Ultra fast non-blocking OOP wrapper for curl_multi_* functions.

Go to Download

A simple and lightweight cURL library with support for asynchronous requests.

Go to Download