Libraries tagged by ai crawler

LLMs.txt Generator - Generates llms.txt files for AI/LLM crawlers with website content in Markdown format, with optional API key protection.

Go to Download

Protect your Laravel app from AI scrapers, LLM crawlers, and prompt injection attacks

Go to Download

Laravel package for smking AEO — auto-inject JSON-LD, FAQ, and AI summary into Laravel responses so AI crawlers (ChatGPT, Perplexity, Google AI) can cite your pages.

Go to Download

Bot and AI-crawler control presets for the Phirewall PHP firewall - block AI scrapers and rate-limit aggressive SEO crawlers

Go to Download

Magento 2 AEO (Answer Engine Optimization) Audit. v4 covers 20 signals across two layers: configuration checks (robots.txt with purpose-classified AI bots, llms.txt + llms.jsonl, llms.txt v2 link relations, agents.md, Product / Organization / FAQ schema, merchant return + shipping policies, sitemap.xml, UCP profile, A2A agent card, AI product feed, OG tags, canonical + hreflang, JSON-LD quality, well-known endpoint matrix, Core Web Vitals via CrUX) and an evidence layer (WAF-vs-robots.txt reality check, observed AI crawler activity via GDPR-safe built-in instrumentation or access-log source). Score Trend dashboard, Admin UI, configurable cron, CI-ready CLI, dependency-injected extension points for custom checkers and evidence sources.

Go to Download

HTML assertions for PHPUnit and Pest: check rendered HTML with CSS selectors, never as a string. With PHPStan rules, a Rector rule and a skill for AI coding agents.

Go to Download

Panth LLMs.txt — AI Indexing Engine for Magento 2. Serves structured /llms.txt, /llms-full.txt and /llms.json with a dynamic multi-URL sitemap engine, weighted ranking, auto-generated summaries, product-type and use-case grouping, and cron-based cache warm-up. Built for ChatGPT, Claude, Perplexity, Gemini and future LLM crawlers.

Go to Download

Official, experimental (AI-generated and AI-maintained) Apify API client for PHP: run Actors, manage storages (datasets, key-value stores, request queues), schedules, webhooks and more.

Go to Download

Kirby robots.txt plugin for blocking AI Crawlers and Bots

Go to Download

Serve Markdown versions of your Craft CMS pages to AI crawlers and LLMs via .md URLs, content negotiation, and user-agent detection.

Go to Download

The files crawlers and AI agents read — sitemap.xml, robots.txt, llms.txt and llms-full.txt — generated from your own models and managed from a Filament 5 panel. Pluggable page sources, reciprocal hreflang clusters with x-default, manual sitemap URLs that never shadow a real page, an automatic sitemap index past 50 000 URLs, per-language llms.txt, an editor page for robots.txt and llms.txt, and scheduled generation.

Go to Download

Server-side AI crawler tracking middleware for Laravel applications.

Go to Download

Symfony Generative Engine Optimization kit (not geolocation): AI crawler robots policy via SeoKit, llms.txt, citation index and audit for FrankenPHP, php-fpm and Nginx.

Go to Download

Laravel middleware za blokiranje AI crawlera i tražilica. Auto-registracija, konfigurabilan, podržava Laravel 10/11/12/13.

Go to Download

Magento 2 module for Answer Engine Optimization (AEO). Manages AI crawler rules in robots.txt (OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Google-Extended, ClaudeBot, Claude-SearchBot, Applebot-Extended, CCBot and more) without overwriting your existing configuration, emits IETF Content-Usage, Cloudflare Content-Signal and RSL License directives, and verifies that a crawler is genuine via Web Bot Auth request signatures (RFC 9421) or vendor-published IP ranges.

Go to Download