LLMs.txt Bundle

Build, dump and serve an llms.txt file from your Symfony application.
The PrestaSitemapBundle way: listeners fill it, a controller serves it, a command dumps it.

📖 Documentation

--- ## Table of Contents - [Features](#features) - [Requirements](#requirements) - [Installation](#installation) - [Configuration](#configuration) - [Adding links](#adding-links) - [From an event listener](#from-an-event-listener) - [Lots of links: generators](#lots-of-links-generators) - [From route options](#from-route-options) - [Serving the file](#serving-the-file) - [On the fly](#on-the-fly) - [Dumped](#dumped) - [Discovery](#discovery) - [License](#license) ## Features - **Event driven**: listeners of `LlmsTxtPopulateEvent` add the links, from your repositories, like `SitemapPopulateEvent` - **Route options**: static pages join the file with an `llms_txt` route option, like the `sitemap` one - **On the fly or dumped**: the controller serves the dumped file when there is one, and builds it otherwise - **Flat memory**: links added as generators are read one at a time, streamed to the response or the dumped file - **Spec compliant**: H1 title, blockquote summary, H2 sections of links, the `Optional` section always last, escaped links - **Discovery**: a `` tag for the `` of your pages and an opt-in `Link` header, through [WebLink](https://symfony.com/doc/current/web_link.html) ## Requirements - PHP 8.2+ - Symfony 6.4, 7.x or 8.x - `symfony/console` for the `llms-txt:dump` command, `symfony/twig-bundle` for the Twig functions ## Installation Register the bundle if Flex did not: Import the route serving `/llms.txt`: ## Configuration ## Adding links ### From an event listener The document can also be changed as a whole: `setTitle()`, `setSummary()`, `setDetails()`, or `section('Projets')->addLink(new Link(...))`. Sections keep the order of their first use, except `Optional` ([its links can be skipped](https://llmstxt.org/#format) when a shorter context is needed), which always comes last. > [!TIP] > Keep the file consistent with your sitemap: add the pages you index, not the `noindex` ones. ### Lots of links: generators `addLink()` keeps each link in memory until the file is rendered. For large sections, hand `addLinks()` an iterable instead: it is only read while the file is rendered, one link at a time, and each line goes to the buffer of the response or to the dumped file as soon as it is rendered. With a generator, the memory stays flat whatever the number of links (200,000 links, a 17 MB file: about 1 MB of memory, against about 100 MB when they are all held). Read the rows by pages too. Doctrine's `toIterable()` does not keep the memory flat: it hydrates one row at a time, but `pdo_mysql` buffers the whole result set client-side, and the entity manager keeps every entity it hydrated. [silarhi/cursor-pagination](https://github.com/silarhi/cursor-pagination) reads them by keyset pages instead, each query starting after the last row of the previous one; selecting scalar fields skips the hydration and the identity map: The generator runs after your listener returns, while the file is rendered: it can be read once, so a document is rendered once. The title is checked before any link is read: a missing title fails with an error page or a failing command, never with a truncated file. ### From route options `section` defaults to `route_section`. An empty title or section fails as soon as the routes load. Routes declared in YAML or XML take the same keys as an array: Only routes without mandatory parameters can carry the option: add the others from a listener. ## Serving the file ### On the fly With the route imported, `GET /llms.txt` builds the file on each request, with a public `Cache-Control` of `max_age` seconds. The file is rendered whole before the response starts, into a `php://temp` buffer that spills to a temporary file past 256 KiB, so the memory stays flat and: - a listener failing half way gives an error page, never a truncated `200` left in the caches - the response carries a `Content-Length` and an `ETag`, and answers `304 Not Modified` to a matching `If-None-Match` ### Dumped The file is streamed to a temporary file next to it, renamed once complete: the web server never serves a half written file, and a failing listener leaves the previous one in place. - **Into `public/`** (the default): the web server serves it without booting Symfony. The route is a fallback until the first dump: remember to dump again, from a cron or after a deployment, or the file goes stale. - **Elsewhere** (`dump_directory: '%kernel.project_dir%/var/llms_txt'`): the controller serves the dumped file when there is one, with `ETag` and `Last-Modified`, and builds it on the fly otherwise. ## Discovery The [llms.txt proposal](https://llmstxt.org) only sets the location of the file, `/llms.txt`. Three ways to advertise it, to combine as you like: `link_header` goes through [WebLink](https://symfony.com/doc/current/web_link.html): the link joins the `_links` of the request, and WebLink writes them all in one `Link` header, next to your `preload()` and `preconnect()` ones. It needs `framework.web_link`, enabled by default when `symfony/web-link` is installed (a dependency of the bundle). The header is added to successful HTML responses of main requests only: not to sub-requests, Turbo Frames, XHR, redirects, errors, JSON or the `llms.txt` file itself. `llms_txt_url()` returns the absolute URL alone. Both use the route of the bundle when it is imported, and `/llms.txt` at the root of the site otherwise. ## License Released under the [MIT License](LICENSE).