Libraries tagged by robotstxt

User-Agent parser for robot rule sets

Go to Download

Sitemap Locator - Looks up XML sitemaps as part of a configured site. Supports various sitemap providers, e.g. by configured page type or robots.txt, and allows to implement custom providers.

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

A modern, fluent PHP package for managing robots.txt rules with type safety and great developer experience

Go to Download

Module for generate robots.txt file by url rules

Go to Download

A PHP library to parse and analyze robots.txt files with validation and URL fetching capabilities

Go to Download

Laravel robots txt package

Go to Download

A library to build robots.txt

Go to Download

Adds a Robots.txt file that is configurable from /admin/settings/.

Go to Download

Kirby Plugin for a best-practice HTML Head Element extendable with snippets.

Go to Download

Professional SEO/AI indexing + Sitemap (index) + RSS + News/Image/Video sitemaps for CodeIgniter 4

Go to Download

Botly is a Filament plugin to manage your site's robots.txt file directly from a Filament admin panel.

Go to Download

Magento 2 AEO (AI Engine Optimization) audit — scores your store across 15 technical signals for AI search and agentic commerce: robots.txt AI-bot policy, llms.txt, llms.jsonl, JSON-LD (Product, Organization, FAQ), merchant return + shipping policies, sitemap.xml, UCP profile, AI product feed, Open Graph, canonical + hreflang, JSON-LD quality, .well-known endpoint matrix and Core Web Vitals via CrUX. Includes a Score Trend dashboard, admin UI, cron scheduling, dynamic fix commands and a dependency-injected extension point for custom checkers.

Go to Download

SEO building blocks for Waaseyaa (sitemap, robots.txt, meta tags, JSON-LD, Twig helpers)

Go to Download

Add filterable attributes to robots.txt in Magento 2

Go to Download