Libraries tagged by 64Robots

Laravel 5.x helpers for a semi-automated robots.txt and meta tags

Go to Download

Class to check a URL for robots exclusion using all possible methods of robots exclusion

Go to Download

Panth Robots SEO — dedicated robots.txt, X-Robots-Tag, and LLM-bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) policy control for Magento 2. Extracted from Panth_AdvancedSEO. Self-contained: emits a per-store robots.txt, adds X-Robots-Tag response headers, validates robots meta directives, and exposes admin CRUD for user-agent / path policies. Hyva and Luma compatible.

Go to Download

A comprehensive PHP package for parsing robots.txt files, including support for meta tags and X-Robots-Tag HTTP headers

Go to Download

Generator robots.txt

Go to Download

Complete SEO Robots solution for Magento 2 with flexible directive management, multiselect UI, and independent X-Robots-Tag configuration

Go to Download

Core implementation of SEO Robots module with flexible directive management and automatic migration

Go to Download

Force robots.txt to disallow all

Go to Download

Simple robots generation module for Silverstripe.

Go to Download

Channel defined robots.txt plugin for Sylius.

Go to Download

Manage your robots.txt file from the WordPress admin

Go to Download

Robots.txt generator

Go to Download

Magento 2 module for AI Engine Optimization (AEO). Manages AI crawler rules in robots.txt (OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Google-Extended, ClaudeBot, Claude-SearchBot, Applebot-Extended, CCBot and more) without overwriting your existing configuration, emits IETF Content-Usage, Cloudflare Content-Signal and RSL License directives, and verifies that a crawler is genuine via Web Bot Auth request signatures (RFC 9421) or vendor-published IP ranges.

Go to Download

Parser for robots.txt files

Go to Download

Robots.txt parser and generator toolset

Go to Download