Libraries tagged by robotstxt
mauricerenck/darkvisitors
280 Downloads
Kirby robots.txt plugin for blocking AI Crawlers and Bots
mad-web/laravel-robots
126899 Downloads
Robots.txt generator service.
fifty-deg/sylius-robots-plugin
8322 Downloads
Channel defined robots.txt plugin for Sylius.
devgeniem/wp-noindex-testing-staging-robots
13284 Downloads
Uses robots.txt to not index development/testing environments of this site with search-engines
arcanedev/robots
614 Downloads
Robots.txt generator
anomaly/robots-extension
41339 Downloads
A simple robots.txt generator.
ellisthedev/laravel-5-robots
214357 Downloads
Robots.txt Generator for Laravel 5
brittainmedia/phpcrawl
3496 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
owenmelbz/laravel-robots-txt
1107 Downloads
Laravel 5.x helpers for a semi-automated robots.txt and meta tags
webignition/robots-txt-parser
16818 Downloads
Parser for robots.txt files
bee4/robots.txt
9333 Downloads
Robots.txt parser and matcher
chrisguitarguy/wp-robots-txt
376 Downloads
Manage your robots.txt file from the WordPress admin
cgoit/contao-robots-txt-bundle
29 Downloads
Gives you the option to use a centralized robots.txt configuration for multiple contao installations.
pixeo/laravel-robots
3173 Downloads
Set the robots.txt content dynamically based on the Laravel app environment.
mkcg/php-query-model
1199 Downloads
Agnostic model to efficiently query and scroll any kind of data (SQL, Search engine, HTTP API, CSV, ...) and push them anywhere with a ETL