Libraries tagged by robots-txt
mad-web/laravel-robots
126811 Downloads
Robots.txt generator service.
glicer/link-checker
11510 Downloads
Check broken links in html files, sitemap.xml and robots.txt
cuab/phpcrawl
32280 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
execut/yii2-robots-txt
11375 Downloads
Module for generate robots.txt file by url rules
evheniy/robots-txt-bundle
14905 Downloads
The RobotsTxtBundle adds the ability to generate robots.txt file to your application.
becklyn/robots-txt
6279 Downloads
A library to build robots.txt
merkandi/laravel-5-robots
7203 Downloads
Robots.txt Generator for Laravel 5
brittainmedia/phpcrawl
2937 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
odorisioe/silverstripe-robots-txt
2409 Downloads
Feature to manage the robots.txt on SilverStripe
innoweb/silverstripe-robots
9028 Downloads
Adds a Robots.txt file that is configurable from /admin/settings/.
bnomei/kirby3-htmlhead
3289 Downloads
Kirby 3 Plugin for a best-practice HTML Head Element extendable with snippets.
mwdelaney/add-sitemap-to-robots
9089 Downloads
Add typical XML sitemap to robots.txt
mtconsultingroup/laravel-robots
2549 Downloads
Robots.txt generator service.
mmerian/phpcrawl
7092 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
fifty-deg/sylius-robots-plugin
2236 Downloads
Channel defined robots.txt plugin for Sylius.