Headless manager for the well-known files every modern web app owes bots AND agents — robots.txt, .well-known/security.txt, llms.txt, humans.txt, sitemap.xml, and an agents/AI manifest. Define them once in code/config and serve them consistently. Zero runtime deps; ships a default-open robots evaluator so scrapers can honor robots honestly, and makes it hard to leak a protected path. Framework-agnostic core + Laravel adapter.

Go to Download

crwlr/crawler-ext-browser

0 Favers
1664 Downloads

Extension for the crwlr/crawler package containing steps utilizing a headless browser.

Go to Download

particle-academy/fancy-seo

1 Favers
142 Downloads

Server-rendered SEO + crawlability for Laravel + Inertia apps — per-route meta/Open Graph/Twitter/JSON-LD on the first byte, dynamic sitemap.xml / robots.txt / llms.txt, and per-page Markdown. The PHP baseline that pairs with @particle-academy/fancy-inertia's .

Go to Download

nitsan/ns-twitter

3 Favers
25039 Downloads

Display fully customizable, responsive, and search engine crawlable Twitter feeds on your TYPO3 website. The TYPO3 Twitter Extension easily integrates to display Tweets that perfectly match your site's appearance and offers many customization options. Easily embed Twitter Timelines, Feeds, Follow buttons, and Like buttons on posts, pages, and widget areas. Documentation & Free Support: https://t3planet.com/typo3-twitter-extension

Go to Download

jooservices/xflickr-crawler

0 Favers
269 Downloads

Standalone Laravel package for queued Flickr crawling with rate limiting and MySQL persistence.

Go to Download

su-sws/nobots

1 Favers
44075 Downloads

This module blocks (well-behaved) search engine robots from crawling, indexing, or archiving your site by setting a "X-Robots-Tag: noindex,nofollow,noarchive" HTTP header.

Go to Download