Libraries tagged by SPiD

CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

Go to Download

A Laravel package to detect web crawlers via the user agent

Go to Download

Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫)，基于phpQuery。

Go to Download

A configurable and extensible PHP web spider

Go to Download

A web crawler or spider library based on Symfony components

Go to Download

Robots.txt parsing library, with full support for every directive and specification.

Go to Download

PHP Wrapper for Curl

Go to Download

wrap curl client (http client) for PHP 5.3; using php multi curl, parallel request and write asynchronous code

Go to Download

A simple and lightweight cURL library with support for asynchronous requests.

Go to Download

This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.

Go to Download

A crawler to find all unique internal pages on a given website

Go to Download

User-Agent parser for robot rule sets

Go to Download

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性

Go to Download

Ultra fast non-blocking OOP wrapper for curl_multi_* functions.

Go to Download

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Go to Download