Download the PHP package rtfirst/llms-txt without Composer

TYPO3 Extension: rt_llms_txt

Generates llms.txt for AI/LLM crawlers - a compact index of your website with SEO metadata and instructions for accessing page content in any language. Optionally protect access with an API key.

Note: This extension implements the llmstxt.org specification.

Concept

The extension provides a two-tier approach for LLM content access:

1. llms.txt - A single index file containing:

   • Website metadata (title, description, domain)
   • Page structure with SEO descriptions and keywords
   • Instructions for accessing full page content
2. Content Format - Access page content via (spec-compliant with llmstxt.org):
   • .md suffix - Clean Markdown (e.g., /page.md)
   • Enabled by default, can be turned off per site with llmsTxt.enableMarkdown

Multi-Language Support

Every enabled language of a site has its own llms.txt below its base:

• Default language: https://example.com/llms.txt
• English with the base /en/: https://example.com/en/llms.txt
• A language with its own domain: https://example.co.uk/llms.txt

Each file lists the pages in its language, with translated titles, descriptions and URLs (with fallbackType: strict, untranslated pages are left out). A ## Languages section links to the llms.txt of the other languages, so crawlers find all of them from any file. A language without pages has no llms.txt (404).

The intro of each language is set in the site configuration (see Site Languages). Page content in any language is available with the .md suffix:

• Default: https://example.com/about.md
• English: https://example.com/en/about.md

Features

• Automatic generation of llms.txt when TYPO3 cache is cleared or the llms.txt site settings change
• One llms.txt per language, linked to each other
• Page properties tab: Configure LLM-specific metadata for each page
• HTML header link: Adds <link rel="alternate"> to HTML pages
• Clean output formats: Well-formatted HTML and Markdown without excessive whitespace
• Flexible configuration: Via Site Settings and page properties

Requirements

• TYPO3 13.0 - 14.x
• PHP 8.2+

Installation

Then activate the extension:

Classic mode (without Composer)

Install the extension from the TER via the Extension Manager, or upload rt_llms_txt_<version>.zip from the assets of a GitHub release. These packages bundle the required library league/html-to-markdown (MIT license) in Resources/Private/Php/ComposerVendor/. The "Source code" archives of GitHub releases do not.

Configuration

Site Settings

Add the Site Set "LLMs.txt Generator" to your site configuration, then configure in Site Settings:

Site Languages

In the Sites module, each language of a site has this field:

Page Properties (LLM Tab)

Each page has an "LLM" tab with these fields:

Output File

llms.txt is not written to public/. The extension serves it dynamically at /llms.txt of each site and below the base of every further language (e.g. /en/llms.txt), and caches the generated content until the TYPO3 cache is flushed or the llmsTxt.* site settings or the site languages change. A static public/llms.txt file would be delivered by the web server instead, so do not create one.

Content Access Formats

Markdown (.md suffix)

Returns clean Markdown with YAML frontmatter. Spec-compliant with llmstxt.org.

Output:

Accessing Different Languages

Simply use the language prefix with the .md suffix:

API Key Protection

You can protect both /llms.txt and the .md suffix endpoint with an API key. This is useful when you want to:

• Restrict access to your own chatbots/RAG systems
• Prevent external scraping of structured content
• Control who can access your LLM-optimized content

Configuration

Set the llmsTxt.apiKey in your Site Settings. Leave empty for public access (default).

Usage

Pass the API key via HTTP header (recommended):

Or via query parameter:

n8n Integration

In n8n HTTP Request node, add the header:

Error Response

Invalid or missing API key returns 401 Unauthorized:

Example llms.txt Output

Each page is one line - [Title](url): notes as defined by llmstxt.org. The notes contain the description, summary and keywords of the page and the link to its Markdown version, so the file can be read by llms.txt parsers. Pages of type "Link" and "Shortcut" only point to another page or URL and have no content of their own, so they are listed without the Markdown link. The ## Languages section links to the llms.txt of the other languages; it is omitted on single-language sites. With llmsTxt.enableMarkdown disabled, the Markdown hints and the [Markdown](…) links are omitted.

robots.txt Configuration

Add these lines to your public/robots.txt to allow AI crawlers:

Add the llms.txt of further languages as well, e.g. Allow: /en/llms.txt.

HTML Header Link

The extension automatically adds a link tag to all HTML pages:

This helps AI crawlers discover the llms.txt file from any page. The link points to the llms.txt of the page language, e.g. /en/llms.txt on English pages.

Development

Code Quality

Testing

CI Pipeline

The extension includes a GitHub Actions workflow (.github/workflows/ci.yaml) that runs for pushes to and pull requests against main and develop:

• PHP-CS-Fixer (code style)
• PHPStan Level 8 (static analysis, TYPO3 13)
• Rector (code modernization)
• Unit and functional tests (PHP 8.2-8.5, TYPO3 13 & 14)
• TER artefact check (classic mode, TYPO3 13 & 14)

Publishing to the TER only happens for version tags (.github/workflows/ter-publish.yaml), which run the same checks first.

Author

Roland Tfirst Email: [email protected]

License

GPL-2.0-or-later