Libraries tagged by robots-txt
glicer/link-checker
12754 Downloads
Check broken links in html files, sitemap.xml and robots.txt
cuab/phpcrawl
33822 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
execut/yii2-robots-txt
11923 Downloads
Module for generate robots.txt file by url rules
evheniy/robots-txt-bundle
17019 Downloads
The RobotsTxtBundle adds the ability to generate robots.txt file to your application.
bnomei/kirby3-htmlhead
3376 Downloads
Kirby 3 Plugin for a best-practice HTML Head Element extendable with snippets.
mmerian/phpcrawl
7111 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
merkandi/laravel-5-robots
10423 Downloads
Robots.txt Generator for Laravel 5
mauricerenck/darkvisitors
108 Downloads
Kirby robots.txt plugin for blocking AI Crawlers and Bots
fifty-deg/sylius-robots-plugin
4772 Downloads
Channel defined robots.txt plugin for Sylius.
gverschuur/laravel-robotstxt
49 Downloads
Set the robots.txt content dynamically based on the Laravel app environment.
monsieurbiz/sylius-robots-txt-plugin
832 Downloads
Manage your robot.txt from your Sylius admin pannel
fuelviews/laravel-robots-txt
200 Downloads
Laravel robots txt package
becklyn/robots-txt
7527 Downloads
A library to build robots.txt
innoweb/silverstripe-robots
10214 Downloads
Adds a Robots.txt file that is configurable from /admin/settings/.
phrozenbyte/pico-robots
805 Downloads
This is Pico's official robots plugin to add a robots.txt and sitemap.xml to your website. Pico is a stupidly simple, blazing fast, flat file CMS.