Libraries tagged by ai crawlers

LLMs.txt Generator - Generates llms.txt files for AI/LLM crawlers with website content in Markdown format, with optional API key protection.

Go to Download

Kirby robots.txt plugin for blocking AI Crawlers and Bots

Go to Download

Serve Markdown versions of your Craft CMS pages to AI crawlers and LLMs via .md URLs, content negotiation, and user-agent detection.

Go to Download

Ban AI crawlers from visiting your Symfony website or application

Go to Download

Laravel package for smking AEO — auto-inject JSON-LD, FAQ, and AI summary into Laravel responses so AI crawlers (ChatGPT, Perplexity, Google AI) can cite your pages.

Go to Download

Server-side AI crawler & bot traffic tracking for Laravel, reporting to DataFast. Unofficial PHP port of @datafast/ai-crawl.

Go to Download

Laravel middleware za blokiranje AI crawlera i tražilica. Auto-registracija, konfigurabilan, podržava Laravel 10/11/12/13.

Go to Download

Protect your Laravel app from AI scrapers, LLM crawlers, and prompt injection attacks

Go to Download

Official PHP client for the WebScraping.AI API — LLM-powered web scraping with rotating proxies and Chromium JavaScript rendering.

Go to Download

Bot and AI-crawler control presets for the Phirewall PHP firewall - block AI scrapers and rate-limit aggressive SEO crawlers

Go to Download

Magento 2 module for AI Engine Optimization (AEO). Injects AI crawler rules (OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, ClaudeBot, anthropic-ai, Claude-User, Applebot, cohere-ai, Amazonbot, Meta-ExternalAgent) into robots.txt — without overwriting your existing configuration. Supports per-bot Allow/Disallow lists, Crawl-delay, Sitemap directives, multi-store, and a public Api\RobotsStatusInterface for cross-module integration with angeo/module-aeo-audit.

Go to Download

Panth LLMs.txt — AI Indexing Engine for Magento 2. Serves structured /llms.txt, /llms-full.txt and /llms.json with a dynamic multi-URL sitemap engine, weighted ranking, auto-generated summaries, product-type and use-case grouping, and cron-based cache warm-up. Built for ChatGPT, Claude, Perplexity, Gemini and future LLM crawlers.

Go to Download

Official, experimental (AI-generated and AI-maintained) Apify API client for PHP: run Actors, manage storages (datasets, key-value stores, request queues), schedules, webhooks and more.

Go to Download

Marked Down serves Markdown to AI crawlers, command-line tools, and APIs while browsers get HTML. Zero-config per-section setup, automatic AI bot detection, .md URL suffix, /llms.txt, YAML front matter, and dedicated Markdown templates — out of the box.

Go to Download

Generate product feed for AI Crawlers

Go to Download