Libraries tagged by robots-txt
mtconsultingroup/laravel-robots
4404 Downloads
Robots.txt generator service.
mmerian/phpcrawl
7118 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
mauricerenck/darkvisitors
255 Downloads
Kirby robots.txt plugin for blocking AI Crawlers and Bots
mad-web/laravel-robots
126896 Downloads
Robots.txt generator service.
eliashaeussler/typo3-sitemap-robots
4566 Downloads
Extension for TYPO3 CMS to enhance robots.txt with sitemap configurations
devgeniem/wp-noindex-testing-staging-robots
12888 Downloads
Uses robots.txt to not index development/testing environments of this site with search-engines
anomaly/robots-extension
40594 Downloads
A simple robots.txt generator.
ellisthedev/laravel-5-robots
214357 Downloads
Robots.txt Generator for Laravel 5
brittainmedia/phpcrawl
3491 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
thelia/edit-robot-txt-module
2349 Downloads
owenmelbz/laravel-robots-txt
1105 Downloads
Laravel 5.x helpers for a semi-automated robots.txt and meta tags
odorisioe/silverstripe-robots-txt
2602 Downloads
Feature to manage the robots.txt on SilverStripe
innmind/robots-txt
420 Downloads
Robots.txt parser
chrisguitarguy/wp-robots-txt
376 Downloads
Manage your robots.txt file from the WordPress admin
webignition/robots-txt-parser
16814 Downloads
Parser for robots.txt files