Libraries tagged by robotstxt
davidpersson/indexed
1154 Downloads
Increase and control index-ability of your site; PHP library for generating sitemaps and controlling all the robots.
andykillen/wordpress-deployment
15 Downloads
Deploy Wordpress via composer with extra tools to make it easier.
urbanstudio/phpcrawl
4 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
svetamor/robots_component
2 Downloads
create a file robots.txt
schnti/robots
471 Downloads
simple kirby robots.txt plugin
sagsoz06/laravel-5-robots
54 Downloads
Robots.txt Generator for Laravel 5
oxyagency/laravel-5-robots
4 Downloads
Robots.txt Generator for Laravel 5
nyx-solutions/yii2-nyx-robots
73 Downloads
Yii2 Robots.txt Generator Module
michaeljjames/silverstripe-robots
98 Downloads
Adds a Robots.txt file that is configurable from /admin/settings/.
loveduckie/silverstripe-robots
1 Downloads
A Silverstripe module for managing the robots.txt file from the CMS.
kazak71/laravel-robots
2 Downloads
Robots.txt generator service.
iteracode/robots
3 Downloads
Robots.txt Shell Generator
ganey/robots
21 Downloads
Robots.txt Generator for Laravel 5
dr4g0nsr/sitemap-crawler
21 Downloads
Crawler for any type of site using robots.txt and sitemap.xml as the source of URL. Useful for cache regenerating.
dawid-z/phpcrawl
88 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.