Libraries tagged by robots-txt

Robots.txt parser and matcher

Go to Download

Module for automatically generating and edit the robots.txt file

Go to Download

Set the robots.txt content dynamically based on the Laravel app environment.

Go to Download

This is Pico's official robots plugin to add a robots.txt and sitemap.xml to your website. Pico is a stupidly simple, blazing fast, flat file CMS.

Go to Download

Class to check a URL for robots exclusion using all possible methods of robots exclusion

Go to Download

Agnostic model to efficiently query and scroll any kind of data (SQL, Search engine, HTTP API, CSV, ...) and push them anywhere with a ETL

Go to Download

parser/handler for Robots Exclusion Protocol (robots.txt and more)

Go to Download

Robots.txt parser and generator toolset

Go to Download

Bundle to create easy XML sitemaps and robots.txt files

Go to Download

simple kirby robots.txt plugin

Go to Download

Opinionated SEO plugin for Kirby 4. With meta description, open graph, sitemap and robots.txt

Go to Download

This module adds the dynamic generation of the robot.txt file to a tool and per site

Go to Download

Crawler for any type of site using robots.txt and sitemap.xml as the source of URL. Useful for cache regenerating.

Go to Download

Robots.txt Generator for Laravel 5

Go to Download

Robots.txt generator

Go to Download