Libraries tagged by robotstxt
glicer/link-checker
12426 Downloads
Check broken links in html files, sitemap.xml and robots.txt
cuab/phpcrawl
33322 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
execut/yii2-robots-txt
11791 Downloads
Module for generate robots.txt file by url rules
evheniy/robots-txt-bundle
16405 Downloads
The RobotsTxtBundle adds the ability to generate robots.txt file to your application.
monsieurbiz/sylius-robots-txt-plugin
583 Downloads
Manage your robot.txt from your Sylius admin pannel
merkandi/laravel-5-robots
9392 Downloads
Robots.txt Generator for Laravel 5
fifty-deg/sylius-robots-plugin
4250 Downloads
Channel defined robots.txt plugin for Sylius.
brittainmedia/phpcrawl
3468 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
gverschuur/laravel-robotstxt
47 Downloads
Set the robots.txt content dynamically based on the Laravel app environment.
becklyn/robots-txt
7246 Downloads
A library to build robots.txt
innoweb/silverstripe-robots
9790 Downloads
Adds a Robots.txt file that is configurable from /admin/settings/.
webvpf/wn-robots-plugin
304 Downloads
Robots - plugin for Winter CMS. Adds robots.txt for your site.
mtconsultingroup/laravel-robots
3331 Downloads
Robots.txt generator service.
mmerian/phpcrawl
7103 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
mauricerenck/darkvisitors
49 Downloads
Kirby robots.txt plugin for blocking AI Crawlers and Bots