Libraries tagged by crawler detector

High-performance device detection engines for the 51Degrees Pipeline API. Accurately identify devices, operating systems, browsers, and crawlers from HTTP headers.

Go to Download

Serve Markdown versions of your Craft CMS pages to AI crawlers and LLMs via .md URLs, content negotiation, and user-agent detection.

Go to Download

Enhanced user agent parser for Laravel

Go to Download

A professional, production-ready PHP XML sitemap generator with advanced features: concurrency, robots.txt, caching, database storage, change detection, SEO analysis, JavaScript rendering, proxy support, webhooks, and more.

Go to Download

Wrapper for the crawler detection package

Go to Download

Craft plugin for detecting bots/crawlers/spiders

Go to Download

Go to Download

Generic Scrapy-like web crawler for Laravel — auto-detects lists, pagination, fields, and supports API/SPA, infinite scroll, image downloading, dedup, resumable crawls, and pluggable output sinks.

Go to Download

This bundle provides console command that detects broken links on your website.

Go to Download

Detect robots and crawlers.

Go to Download

Protect your Laravel app from AI scrapers, LLM crawlers, and prompt injection attacks

Go to Download

Marked Down serves Markdown to AI crawlers, command-line tools, and APIs while browsers get HTML. Zero-config per-section setup, automatic AI bot detection, .md URL suffix, /llms.txt, YAML front matter, and dedicated Markdown templates — out of the box.

Go to Download

Go to Download

A Symfony 2 crawler detect bundle

Go to Download

Directory Crawler PHP is a simple PHP library for recursively crawling through directories and listing files and directories.

Go to Download