Libraries tagged by robotstxt

Set the robots.txt content dynamically based on the Laravel app environment.

Go to Download

Class to check a URL for robots exclusion using all possible methods of robots exclusion

Go to Download

Agnostic model to efficiently query and scroll any kind of data (SQL, Search engine, HTTP API, CSV, ...) and push them anywhere with a ETL

Go to Download

Generator robots.txt

Go to Download

Generate sitemap.xml and robots.txt files in Laravel or any PHP Application with ease.

Go to Download

parser/handler for Robots Exclusion Protocol (robots.txt and more)

Go to Download

Robots.txt parser and generator toolset

Go to Download

This is Pico's official robots plugin to add a robots.txt and sitemap.xml to your website. Pico is a stupidly simple, blazing fast, flat file CMS.

Go to Download

Generate sitemap.xml with respect to robots.txt disallow rules, according specification https://www.sitemaps.org and without dependencies from any framework or cms.

Go to Download

simple kirby robots.txt plugin

Go to Download

Opinionated SEO plugin for Kirby 4. With meta description, open graph, sitemap and robots.txt

Go to Download

Yii2 Robots.txt Generator Module

Go to Download

Crawler for any type of site using robots.txt and sitemap.xml as the source of URL. Useful for cache regenerating.

Go to Download

Robots.txt generator

Go to Download

Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...

Go to Download