Libraries tagged by robotstxt

Increase and control index-ability of your site; PHP library for generating sitemaps and controlling all the robots.

Go to Download

Deploy Wordpress via composer with extra tools to make it easier.

Go to Download

X-Robots-Tag HTTP header parser class

Go to Download

Clean-Param URL filtering class - Find and remove any duplicate URL

Go to Download

AI Agent Readiness for Magento 2 / Adobe Commerce: the eight public agent-readiness checks run in-admin against your own store, plus a counter of AI agent traffic (GPTBot, ClaudeBot, PerplexityBot, ...).

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...

Go to Download

create a file robots.txt

Go to Download

Lightweight, framework-agnostic robots.txt management for PHP

Go to Download

simple kirby robots.txt plugin

Go to Download

Robots.txt Generator for Laravel 5

Go to Download

Domain monitoring toolkit for HTTP, SSL, WHOIS, DNS, ports, security headers, robots.txt, and sitemaps.

Go to Download

Laravel package that generates robots.txt, sitemap.xml, and llms.txt — and imports pSEO campaign archives.

Go to Download

Robots.txt Generator for Laravel 5

Go to Download

Automatic sitemap.xml and robots.txt generation from scanned routes for NeoPHP

Go to Download