Libraries tagged by robots.txt
mmerian/phpcrawl
7121 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
mauricerenck/darkvisitors
267 Downloads
Kirby robots.txt plugin for blocking AI Crawlers and Bots
mad-web/laravel-robots
126898 Downloads
Robots.txt generator service.
anomaly/robots-extension
41019 Downloads
A simple robots.txt generator.
ellisthedev/laravel-5-robots
214357 Downloads
Robots.txt Generator for Laravel 5
owenmelbz/laravel-robots-txt
1106 Downloads
Laravel 5.x helpers for a semi-automated robots.txt and meta tags
webignition/robots-txt-parser
16817 Downloads
Parser for robots.txt files
bee4/robots.txt
9333 Downloads
Robots.txt parser and matcher
chrisguitarguy/wp-robots-txt
376 Downloads
Manage your robots.txt file from the WordPress admin
oihana/php-robots
6 Downloads
The Oihana PHP Robots library
nickmoline/robots-checker
858 Downloads
Class to check a URL for robots exclusion using all possible methods of robots exclusion
mkcg/php-query-model
1199 Downloads
Agnostic model to efficiently query and scroll any kind of data (SQL, Search engine, HTTP API, CSV, ...) and push them anywhere with a ETL
jazzman/php-robots
235 Downloads
Generator robots.txt
hettiger/seo-aggregator
371 Downloads
Generate sitemap.xml and robots.txt files in Laravel or any PHP Application with ease.
diggin/diggin-robotrules
187 Downloads
parser/handler for Robots Exclusion Protocol (robots.txt and more)