Libraries tagged by robots-txt
davidpersson/indexed
1156 Downloads
Increase and control index-ability of your site; PHP library for generating sitemaps and controlling all the robots.
andykillen/wordpress-deployment
15 Downloads
Deploy Wordpress via composer with extra tools to make it easier.
webarchitect609/sitemap
1030 Downloads
Generate sitemap.xml with respect to robots.txt disallow rules, according specification https://www.sitemaps.org and without dependencies from any framework or cms.
urbanstudio/phpcrawl
4 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
twentytwo-labs/behat-seo-extension
105 Downloads
Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...
svetamor/robots_component
2 Downloads
create a file robots.txt
sagsoz06/laravel-5-robots
54 Downloads
Robots.txt Generator for Laravel 5
oxyagency/laravel-5-robots
4 Downloads
Robots.txt Generator for Laravel 5
nyx-solutions/yii2-nyx-robots
83 Downloads
Yii2 Robots.txt Generator Module
mwdelaney/add-sitemap-to-robots
9144 Downloads
Add typical XML sitemap to robots.txt
monsieurbiz/mbiz_robotstxt
152 Downloads
Magento extension to edit robots.txt from the admin panel.
michaeljjames/silverstripe-robots
98 Downloads
Adds a Robots.txt file that is configurable from /admin/settings/.
loveduckie/silverstripe-robots
1 Downloads
A Silverstripe module for managing the robots.txt file from the CMS.
kazak71/laravel-robots
2 Downloads
Robots.txt generator service.
jamesshaver/phpcrawl
15 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.