Libraries tagged by robots.txt

Robots.txt parser and generator toolset

Go to Download

Robots - plugin for Winter CMS. Adds robots.txt for your site.

Go to Download

Generate sitemap.xml with respect to robots.txt disallow rules, according specification https://www.sitemaps.org and without dependencies from any framework or cms.

Go to Download

Bundle to create easy XML sitemaps and robots.txt files

Go to Download

simple kirby robots.txt plugin

Go to Download

Check broken links in html files, sitemap.xml and robots.txt

Go to Download

Crawler for any type of site using robots.txt and sitemap.xml as the source of URL. Useful for cache regenerating.

Go to Download

Uses robots.txt to not index development/testing environments of this site with search-engines

Go to Download

Robots.txt generator

Go to Download

Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...

Go to Download

lib used for parsing a robots.txt file

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

Provides various framework-agnostic ways to generate the contents of the robots.txt file

Go to Download

Kohana module for providing robots.txt file

Go to Download

Generate custom robots.txt for subsites

Go to Download