Libraries tagged by robots.txt

Set the robots.txt content dynamically based on the Laravel app environment.

Go to Download

Class to check a URL for robots exclusion using all possible methods of robots exclusion

Go to Download

Dynamic robots.txt files for Laravel projects

Go to Download

Agnostic model to efficiently query and scroll any kind of data (SQL, Search engine, HTTP API, CSV, ...) and push them anywhere with a ETL

Go to Download

Non Production Robots Ignore is a simple WordPress plugin that is used with vlucas/phpdotenv (or similar) that sets environment variables to set robots.txt to ignore all on all environments except production.

Go to Download

Generate sitemap.xml and robots.txt files in Laravel or any PHP Application with ease.

Go to Download

parser/handler for Robots Exclusion Protocol (robots.txt and more)

Go to Download

Robots.txt parser and generator toolset

Go to Download

Generate sitemap.xml with respect to robots.txt disallow rules, according specification https://www.sitemaps.org and without dependencies from any framework or cms.

Go to Download

Opinionated SEO plugin for Kirby 4. With meta description, open graph, sitemap and robots.txt

Go to Download

Add typical XML sitemap to robots.txt

Go to Download

This module adds the dynamic generation of the robot.txt file to a tool and per site

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

robots.txt generator for BoomCMS

Go to Download

Robots.txt generator

Go to Download