Libraries tagged by crawler detector

Generic Scrapy-like web crawler for Laravel — auto-detects lists, pagination, fields, and supports API/SPA, infinite scroll, image downloading, dedup, resumable crawls, and pluggable output sinks.

Go to Download

Link crawler robot — Composer/Packagist mirror of the Moodle plugin tool_crawler, auto-updated from the Moodle plugins directory.

Go to Download

The standalone PHP Browscap parser Crossjoin\Browscap detects browser properties as well as device information based on the user agent string of the requesting browsers and search engines, using the data from the Browser Capabilities Project. It's several hundred times faster than the build-in PHP function get_browser(), and faster than other Browscap PHP libraries, with much lower memory consumption. Optionally Crossjoin\Browscap automatically updates the Browscap data, so you're always up-to-date. The newest version is build for PHP 7.x, for PHP >= 5.6 use version 2.x, for PHP >= 5.3 use version 1.x.

Go to Download

Contao helper class to detect search engines, bots, spiders, crawlers ...

Go to Download

Spyglasses AI Traffic Analytics for Drupal - Detect, block and log your website's AI traffic using Drupal's middleware API.

Go to Download

A Symfony bundle to crawl a site and detect broken internal and external links.

Go to Download

Traffic Sentinel for Laravel: tracks visitors + bots, online presence, and traffic stats.

Go to Download

A Laravel package that crawls websites to detect broken links, redirect chains, HTTPS downgrades, and more. Includes JavaScript rendering support for SPAs.

Go to Download

Protect your Laravel app from AI scrapers, LLM crawlers, and prompt injection attacks

Go to Download

This module blocks all kind of bots / crawlers / spiders.

Go to Download

The package which identify the Bot or Crawler which help to identify the how many bots or crawler visiting in day and from where

Go to Download

TYPO3 extension that protects the Apache Solr search endpoint from bots and crawlers via URI obfuscation and bot detection.

Go to Download

Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫)，基于phpQuery。

Go to Download

DOM crawler with advanced selector API and assertions.

Go to Download

Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫)，基于phpQuery。

Go to Download