Libraries tagged by robots-txt
cuab/phpcrawl
36723 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
monsieurbiz/sylius-robots-txt-plugin
5544 Downloads
Manage your robot.txt from your Sylius admin pannel
execut/yii2-robots-txt
12496 Downloads
Module for generate robots.txt file by url rules
evheniy/robots-txt-bundle
19738 Downloads
The RobotsTxtBundle adds the ability to generate robots.txt file to your application.
innoweb/silverstripe-robots
13340 Downloads
Adds a Robots.txt file that is configurable from /admin/settings/.
merkandi/laravel-5-robots
18762 Downloads
Robots.txt Generator for Laravel 5
eliashaeussler/typo3-sitemap-robots
6053 Downloads
Extension for TYPO3 CMS to enhance robots.txt with sitemap configurations
thelia/edit-robot-txt-module
2447 Downloads
fuelviews/laravel-robots-txt
1077 Downloads
Laravel robots txt package
fkrzski/robots-txt
871 Downloads
A modern, fluent PHP package for managing robots.txt rules with type safety and great developer experience
becklyn/robots-txt
8381 Downloads
A library to build robots.txt
phrozenbyte/pico-robots
832 Downloads
This is Pico's official robots plugin to add a robots.txt and sitemap.xml to your website. Pico is a stupidly simple, blazing fast, flat file CMS.
gverschuur/laravel-robotstxt
51 Downloads
Set the robots.txt content dynamically based on the Laravel app environment.
mtconsultingroup/laravel-robots
4673 Downloads
Robots.txt generator service.
mmerian/phpcrawl
7121 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.