Libraries tagged by robotstxt

Robots.txt Generator for Laravel 5

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

Polite URL crawler. Transport + politeness only (robots.txt, per-host rate limits, conditional requests). Parsing and consuming content is the caller's job.

Go to Download

Enterprise-grade SEO Tools for CodeIgniter 4 — meta tags, Open Graph, Twitter Cards, JSON-LD schema types, sitemaps, robots.txt, hreflang, AI-bot governance and more.

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

Robots.txt Generator for Laravel 5

Go to Download

WordPress equivalent of drupal_cache_protection. Strips/redirects tracking params so Varnish caches canonical URLs, augments robots.txt with bot-throttling rules, and exposes filters for site-specific extension.

Go to Download

Implements a dynamic robots.txt file

Go to Download

Yii2 Robots.txt Generator Module

Go to Download

Yii2 Robots.txt Generator Module

Go to Download

SEO Kit for Kirby 3 including sitemap.xml and robotx.txt

Go to Download

This module adds the dynamic generation of the robot.txt file to a tool and per site

Go to Download