Libraries tagged by crawler detection

The standalone PHP Browscap parser Crossjoin\Browscap detects browser properties as well as device information based on the user agent string of the requesting browsers and search engines, using the data from the Browser Capabilities Project. It's several hundred times faster than the build-in PHP function get_browser(), and faster than other Browscap PHP libraries, with much lower memory consumption. Optionally Crossjoin\Browscap automatically updates the Browscap data, so you're always up-to-date. The newest version is build for PHP 7.x, for PHP >= 5.6 use version 2.x, for PHP >= 5.3 use version 1.x.

Go to Download

Protect your Laravel app from AI scrapers, LLM crawlers, and prompt injection attacks

Go to Download

A professional, production-ready PHP XML sitemap generator with advanced features: concurrency, robots.txt, caching, database storage, change detection, SEO analysis, JavaScript rendering, proxy support, webhooks, and more.

Go to Download

Contao bundle helper class to detect search engines, bots, spiders, crawlers ...

Go to Download

Magento 2 AEO (Answer Engine Optimization) Audit. v4 covers 20 signals across two layers: configuration checks (robots.txt with purpose-classified AI bots, llms.txt + llms.jsonl, llms.txt v2 link relations, agents.md, Product / Organization / FAQ schema, merchant return + shipping policies, sitemap.xml, UCP profile, A2A agent card, AI product feed, OG tags, canonical + hreflang, JSON-LD quality, well-known endpoint matrix, Core Web Vitals via CrUX) and an evidence layer (WAF-vs-robots.txt reality check, observed AI crawler activity via GDPR-safe built-in instrumentation or access-log source). Score Trend dashboard, Admin UI, configurable cron, CI-ready CLI, dependency-injected extension points for custom checkers and evidence sources.

Go to Download

Serve Markdown versions of your Craft CMS pages to AI crawlers and LLMs via .md URLs, content negotiation, and user-agent detection.

Go to Download

A PHP desktop/mobile user agent parser with bot detection, based on Mobiledetect and CrawlerDetect.

Go to Download

High-performance device detection engines for the 51Degrees Pipeline API. Accurately identify devices, operating systems, browsers, and crawlers from HTTP headers.

Go to Download

Wrapper for the crawler detection package

Go to Download

Free cookieless WordPress analytics plugin: real-time visitor stats with humans, bots and AI crawlers counted separately, outbound click tracking, public Stats API and MCP server. Free alternative to Plausible, Fathom, Matomo, MonsterInsights and Google Analytics.

Go to Download

PHP library to block unwanted crawlers, scrapers and bad bots by User-Agent, IP block lists and allow lists. No captcha, works on any PHP site. Self-hosted alternative to Cloudflare bot management.

Go to Download

Contao helper class to detect search engines, bots, spiders, crawlers ...

Go to Download

Marked Down serves Markdown to AI crawlers, command-line tools, and APIs while browsers get HTML. Zero-config per-section setup, automatic AI bot detection, .md URL suffix, /llms.txt, YAML front matter, and dedicated Markdown templates — out of the box.

Go to Download

Report bot and AI-crawler detections to WebDecoy from any PHP application

Go to Download

The standalone PHP Browscap parser Crossjoin\Browscap detects browser properties as well as device information based on the user agent string of the requesting browsers and search engines, using the data from the Browser Capabilities Project. It's several hundred times faster than the build-in PHP function get_browser(), and faster than other Browscap PHP libraries, with much lower memory consumption. Optionally Crossjoin\Browscap automatically updates the Browscap data, so you're always up-to-date. The newest version is build for PHP 7.x, for PHP >= 5.6 use version 2.x, for PHP >= 5.3 use version 1.x.

Go to Download