Libraries tagged by robots-txt

User-Agent parser for robot rule sets

Go to Download

Sitemap Locator - Looks up XML sitemaps as part of a configured site. Supports various sitemap providers, e.g. by configured page type or robots.txt, and allows to implement custom providers.

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

A PHP library to parse and analyze robots.txt files with validation and URL fetching capabilities

Go to Download

Laravel robots txt package

Go to Download

A modern, fluent PHP package for managing robots.txt rules with type safety and great developer experience

Go to Download

Module for generate robots.txt file by url rules

Go to Download

Parser for robots.txt files

Go to Download

Adds a Robots.txt file that is configurable from /admin/settings/.

Go to Download

Professional SEO/AI indexing + Sitemap (index) + RSS + News/Image/Video sitemaps for CodeIgniter 4

Go to Download

Botly is a Filament plugin to manage your site's robots.txt file directly from a Filament admin panel.

Go to Download

Magento 2 AEO (AI Engine Optimization) Audit. v3 covers 15 signals — robots.txt AI bots, llms.txt + llms.jsonl, Product / Organization / FAQ schema, merchant return + shipping policies, sitemap.xml, UCP profile, AI product feed, OG tags, canonical + hreflang, JSON-LD quality, well-known endpoint matrix, Core Web Vitals via CrUX. Score Trend dashboard, Admin UI, cron, dynamic fix commands, dependency-injected extension point for custom checkers.

Go to Download

SEO building blocks for Waaseyaa (sitemap, robots.txt, meta tags, JSON-LD, Twig helpers)

Go to Download

Robots.txt Generator for Laravel 5

Go to Download

Kirby robots.txt plugin for blocking AI Crawlers and Bots

Go to Download