Download the PHP package rkd/module-llms-txt without Composer

RKD LLMs.txt Generator for Magento 2

Generate llms.txt and llms-full.txt files for your Magento 2 store, making your product catalog discoverable by AI assistants like ChatGPT, Claude, and Perplexity.

What is llms.txt?

llms.txt is an emerging standard (like robots.txt for search engines) that helps Large Language Models understand your website content. This module automatically generates structured, AI-optimized files from your Magento catalog.

Features

• llms.txt + llms-full.txt generation (both files, spec-compliant)
• Inventory-aware output: excludes out-of-stock products (SQL-level filtering)
• Variant intelligence: configurable product options (color, size), bundle items, grouped products, customizable options
• Niche-adaptive: admin selects which product attributes to expose (gender for apparel, specs for electronics, ingredients for food)
• Rich product data: price, SKU, descriptions, custom attributes
• Category breadcrumbs: hierarchical category paths with product counts
• Spec validation: checks output against the llms.txt standard
• Built-in sanitization: HTML entities decoded, whitespace normalized, tables converted to markdown
• Multi-store & multi-language: one file set per store view with automatic cross-language discovery — each language's llms.txt links to its siblings so AI crawlers find them all
• Cron-based auto-regeneration with change detection (dirty flags)
• Manual generation: admin button, CLI command, REST API
• Robots.txt integration: auto-injects llms.txt references
• Performance optimized: cursor-based pagination (no OFFSET degradation) + batched EAV queries. Tested on catalogs up to 100K products.

Requirements

• Magento 2.4.7 or later (tested on 2.4.7-p1 and 2.4.8)
• PHP 8.1 or later (developed on PHP 8.3)

Installation

Via Composer (recommended)

Manual Installation

1. Create app/code/RKD/LlmsTxt/ directory
2. Copy module files into it
3. Run:

Configuration

Navigate to Stores > Configuration > RKD > LLMs.txt Generator

General Settings

Content Sections

Product Data

llms-full.txt Settings

Usage

CLI Command

REST API

URLs

After generation, the files are served at:

• https://yourstore.com/llms.txt
• https://yourstore.com/llms-full.txt

Admin Panel

• Marketing > LLMs.txt > Generate Files — trigger generation
• Marketing > LLMs.txt > Preview Output — preview without writing
• Marketing > LLMs.txt > Configuration — module settings

How It Works

1. SectionProviders collect data from Magento (metadata, CMS pages, categories, products)
2. Generator orchestrates providers, builds markdown output
3. Validator checks spec compliance
4. FileWriter writes to var/rkd_llmstxt/ with atomic writes
5. Router serves the files at /llms.txt with proper UTF-8 headers
6. Observers detect entity changes and set dirty flags
7. Cron checks dirty flags and regenerates when needed

Performance

The module uses cursor-based pagination (WHERE entity_id > :lastId — no OFFSET performance cliff) combined with PHP generators (yield) so each batch is processed and released before the next is loaded. Batch-level memory is bounded; total memory scales with catalog size because the final output is assembled before writing. Write-up of the technique: Paginating Magento catalogs without OFFSET.

By default Product Limit is 0 (unlimited) — the full visible catalog is included. You can optionally cap it via Stores > Configuration > RKD > LLMs.txt > Product Limit (for example, to generate a smaller "top products" file). When a configured cap is hit, the generation result surfaces a clear warning telling you how many products were excluded. For very large catalogs (50K+ products), ensure PHP memory_limit is sized accordingly using the table above.

Multi-Store and Multi-Language Support

Multi-language is a first-class feature, not an afterthought. The module generates one file set per store view and makes every language discoverable by AI crawlers from any entry point.

How it works

For each active store view, the module writes:

• var/rkd_llmstxt/{store_code}/llms.txt
• var/rkd_llmstxt/{store_code}/llms-full.txt

…served at the store view's public URL:

Automatic cross-language discovery

Each file includes an "Available in Other Languages" section listing sibling store views on the same website:

An AI that fetches any one language's llms.txt will find the others automatically — no external sitemap or registry needed.

Language handling internals

Content resolution follows Magento's native EAV store-scope fallback:

• Product, category, and CMS fields load at the requested store view's scope
• When a field isn't translated for that store view, the default (admin-scope) value is used — identical to Magento's storefront behavior
• UTF-8 is preserved end-to-end: Cyrillic, Greek, Chinese, Arabic, emoji, and extended Latin (ä, õ, ß, ü) all render correctly

Note: For AI output to be fully localized per language, merchants should translate product names, descriptions, and attribute values for each store view. If content isn't translated, the fallback (default-scope) value appears — accurate to what the storefront serves, but potentially mixing languages in the output. This is a content-translation responsibility, not a module limitation.

Supported Product Types

Change Detection

The module tracks changes via Magento events:

• catalog_product_save_after — product changes
• cataloginventory_stock_item_save_after — stock changes
• cms_page_save_after — CMS page changes
• catalog_category_save_after — category changes

When a change is detected, a dirty flag is set. The next cron run regenerates the files.

License

MIT License. See LICENSE.txt.

Support

• Bug reports and feature requests: GitHub Issues
• Questions and discussion: GitHub Discussions
• Author: Robin Dhiman — senior web engineer, fifteen years in Magento 2 and Hyvä. More writing at devrob.in/blog; contact [email protected] for engagements.