Libraries tagged by robotstxt

parser/handler for Robots Exclusion Protocol (robots.txt and more)

Go to Download

Robots.txt parser and generator toolset

Go to Download

This is Pico's official robots plugin to add a robots.txt and sitemap.xml to your website. Pico is a stupidly simple, blazing fast, flat file CMS.

Go to Download

Robots - plugin for Winter CMS. Adds robots.txt for your site.

Go to Download

Generate sitemap.xml with respect to robots.txt disallow rules, according specification https://www.sitemaps.org and without dependencies from any framework or cms.

Go to Download

Bundle to create easy XML sitemaps and robots.txt files

Go to Download

simple kirby robots.txt plugin

Go to Download

Check broken links in html files, sitemap.xml and robots.txt

Go to Download

Crawler for any type of site using robots.txt and sitemap.xml as the source of URL. Useful for cache regenerating.

Go to Download

Robots.txt generator

Go to Download

Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...

Go to Download

lib used for parsing a robots.txt file

Go to Download

Provides various framework-agnostic ways to generate the contents of the robots.txt file

Go to Download

Kohana module for providing robots.txt file

Go to Download

Generate custom robots.txt for subsites

Go to Download