Libraries tagged by crawler

Symfony Spider Bundle

Go to Download

Crawler + Scraper + Parser library

Go to Download

Crawl, match, parse IP or IP range, check if IP or range is in another range. Support IPv4, IPv6, IP Interval, Wildcard and CIDR. Check if IP is Cloudflare node IP, Google bot IP. 爬取，正则匹配，解析 IP 和 IP 范围，检测 IP 或范围是否在另一个范围中。支持 IPv4，IPv6，区间、通配符或 CIDR 表示的 IP 范围。检测 IP 是否是 Cloudflare 节点或 Google 漫游器 IP

Go to Download

PHP library providing functionality to verify that user-agents are who they claim to be.

Go to Download

Sitemap crawler and generator class

Go to Download

A modern, fluent PHP package for managing robots.txt rules with type safety and great developer experience

Go to Download

The BVP Scraper for Boatrace.

Go to Download

jQuery-inspired DOM manipulation extension for Symfony's Crawler

Go to Download

The standalone PHP Browscap parser Crossjoin\Browscap detects browser properties as well as device information based on the user agent string of the requesting browsers and search engines, using the data from the Browser Capabilities Project. It's several hundred times faster than the build-in PHP function get_browser(), and faster than other Browscap PHP libraries, with much lower memory consumption. Optionally Crossjoin\Browscap automatically updates the Browscap data, so you're always up-to-date. The newest version is build for PHP 7.x, for PHP >= 5.6 use version 2.x, for PHP >= 5.3 use version 1.x.

Go to Download

A simple website crawler & sitemap generator without a headless browser for Laravel 5.8+

Go to Download

A crawler to find all unique internal pages on a given website

Go to Download

Encrypt e-mail addresses and phone numbers to protect against unwanted crawlers & spam.

Go to Download

Yet another DOM crawler, inspired by symfony/dom-crawler

Go to Download

Prevent crawlers and IPs reported from creating a session

Go to Download

Generates an XML site map for use by web crawlers

Go to Download