Libraries tagged by robotstxt
diggin/diggin-robotrules
187 Downloads
parser/handler for Robots Exclusion Protocol (robots.txt and more)
bm1/file-noindex
10 Downloads
Exclude single files/images from search engines (Google Image Search) via a dynamically generated robots.txt - toggled per file in the file list.
glow/robots
1309 Downloads
Robots.txt parser and generator toolset
phrozenbyte/pico-robots
858 Downloads
This is Pico's official robots plugin to add a robots.txt and sitemap.xml to your website. Pico is a stupidly simple, blazing fast, flat file CMS.
mage2kishan/module-robots-seo
30 Downloads
Panth Robots SEO — dedicated robots.txt, X-Robots-Tag, and LLM-bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) policy control for Magento 2. Extracted from Panth_AdvancedSEO. Self-contained: emits a per-store robots.txt, adds X-Robots-Tag response headers, validates robots meta directives, and exposes admin CRUD for user-agent / path policies. Hyva and Luma compatible.
ibrahim-eng12/seoquent
6 Downloads
Seoquent - A powerful all-in-one Laravel SEO package. Meta tags, Open Graph, Twitter Cards, JSON-LD structured data, sitemap generation, robots.txt, and per-model SEO management.
rasuvaeff/domain-monitor
8 Downloads
Domain monitoring toolkit for HTTP, SSL, WHOIS, DNS, ports, security headers, robots.txt, and sitemaps.
presprog/kirby-seo-plugin
555 Downloads
Opinionated SEO plugin for Kirby 4. With meta description, open graph, sitemap and robots.txt
mtconsultingroup/laravel-robots
4802 Downloads
Robots.txt generator service.
glicer/link-checker
13665 Downloads
Check broken links in html files, sitemap.xml and robots.txt
dr4g0nsr/sitemap-crawler
22 Downloads
Crawler for any type of site using robots.txt and sitemap.xml as the source of URL. Useful for cache regenerating.
arcanedev/robots
617 Downloads
Robots.txt generator
mortola/behat-seo-contexts
5290 Downloads
Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...
m6web/roboxt
575 Downloads
lib used for parsing a robots.txt file
brittainmedia/phpcrawl
3504 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.