Libraries tagged by robotstxt

Robots.txt Generator for Laravel 5

Go to Download

Extension for TYPO3 CMS to enhance robots.txt with sitemap configurations

Go to Download

Force robots.txt to disallow all

Go to Download

The RobotsTxtBundle adds the ability to generate robots.txt file to your application.

Go to Download

Magento 2 module for AI Engine Optimization (AEO). Injects AI crawler rules (OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, ClaudeBot, anthropic-ai, Claude-User, Applebot, cohere-ai, Amazonbot, Meta-ExternalAgent) into robots.txt — without overwriting your existing configuration. Supports per-bot Allow/Disallow lists, Crawl-delay, Sitemap directives, multi-store, and a public Api\RobotsStatusInterface for cross-module integration with angeo/module-aeo-audit.

Go to Download

Parser for robots.txt files

Go to Download

Set the robots.txt content dynamically based on the Laravel app environment.

Go to Download

Headless manager for the well-known files every modern web app owes bots AND agents — robots.txt, .well-known/security.txt, llms.txt, humans.txt, sitemap.xml, and an agents/AI manifest. Define them once in code/config and serve them consistently. Zero runtime deps; ships a default-open robots evaluator so scrapers can honor robots honestly, and makes it hard to leak a protected path. Framework-agnostic core + Laravel adapter.

Go to Download

SEO management plugin for Filament v5 - meta tags, global settings, Schema.org, robots.txt

Go to Download

A professional, production-ready PHP XML sitemap generator with advanced features: concurrency, robots.txt, caching, database storage, change detection, SEO analysis, JavaScript rendering, proxy support, webhooks, and more.

Go to Download

Robots - plugin for Winter CMS. Adds robots.txt for your site.

Go to Download

Server-rendered SEO + crawlability for Laravel + Inertia apps — per-route meta/Open Graph/Twitter/JSON-LD on the first byte, dynamic sitemap.xml / robots.txt / llms.txt, and per-page Markdown. The PHP baseline that pairs with @particle-academy/fancy-inertia's .

Go to Download

Symfony bundle for SEO management: meta tags (OpenGraph, Twitter Cards), Schema.org JSON-LD structured data, sitemap XML generation, and robots.txt

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

Kirby robots.txt plugin for blocking AI Crawlers and Bots

Go to Download