Libraries tagged by robots.txt

Agnostic model to efficiently query and scroll any kind of data (SQL, Search engine, HTTP API, CSV, ...) and push them anywhere with a ETL

Go to Download

Generate sitemap.xml and robots.txt files in Laravel or any PHP Application with ease.

Go to Download

parser/handler for Robots Exclusion Protocol (robots.txt and more)

Go to Download

Robots.txt parser and generator toolset

Go to Download

This is Pico's official robots plugin to add a robots.txt and sitemap.xml to your website. Pico is a stupidly simple, blazing fast, flat file CMS.

Go to Download

Robots - plugin for Winter CMS. Adds robots.txt for your site.

Go to Download

Check broken links in html files, sitemap.xml and robots.txt

Go to Download

Crawler for any type of site using robots.txt and sitemap.xml as the source of URL. Useful for cache regenerating.

Go to Download

Robots.txt generator

Go to Download

Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...

Go to Download

lib used for parsing a robots.txt file

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

Provides various framework-agnostic ways to generate the contents of the robots.txt file

Go to Download

Kohana module for providing robots.txt file

Go to Download

Generate custom robots.txt for subsites

Go to Download