Libraries tagged by robotstxt
webarchitect609/sitemap
964 Downloads
Generate sitemap.xml with respect to robots.txt disallow rules, according specification https://www.sitemaps.org and without dependencies from any framework or cms.
urbanstudio/phpcrawl
3 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
svetamor/robots_component
2 Downloads
create a file robots.txt
schnti/robots
471 Downloads
simple kirby robots.txt plugin
sagsoz06/laravel-5-robots
54 Downloads
Robots.txt Generator for Laravel 5
presprog/kirby-seo-plugin
131 Downloads
Opinionated SEO plugin for Kirby 4. With meta description, open graph, sitemap and robots.txt
oxyagency/laravel-5-robots
3 Downloads
Robots.txt Generator for Laravel 5
michaeljjames/silverstripe-robots
98 Downloads
Adds a Robots.txt file that is configurable from /admin/settings/.
kazak71/laravel-robots
2 Downloads
Robots.txt generator service.
jamesshaver/phpcrawl
9 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
iteracode/robots
3 Downloads
Robots.txt Shell Generator
ganey/robots
19 Downloads
Robots.txt Generator for Laravel 5
dr4g0nsr/sitemap-crawler
20 Downloads
Crawler for any type of site using robots.txt and sitemap.xml as the source of URL. Useful for cache regenerating.
dawid-z/phpcrawl
86 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
beeg99/phpcrawl
3 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.