Libraries tagged by robotstxt
m6web/roboxt
573 Downloads
lib used for parsing a robots.txt file
brittainmedia/phpcrawl
3485 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
tekord/robots-txt-provider
3 Downloads
Provides various framework-agnostic ways to generate the contents of the robots.txt file
spotman/kohana-robots.txt
9 Downloads
Kohana module for providing robots.txt file
rotassator/silverstripe-subsites-robotstxt
760 Downloads
Generate custom robots.txt for subsites
ricbra/robots-txt-bundle
5487 Downloads
Simple Symfony2 which generates robots.txt based on config setting
ranvis/robots-txt-processor-test
2 Downloads
robots.txt processor feature/behavior test.
ranvis/robots-txt-processor
21 Downloads
robots.txt filter and tester for untrusted source.
jmajors/robotstxt
21 Downloads
A small package for parsing websites' robots.txt files
cybercog/robots-txt
2400 Downloads
Robots.txt Generator
b1tc0re/robots-txt
38 Downloads
Generator robots.txt
andronof/robotstxt
70 Downloads
Parse robots.txt
itinerisltd/add-yoast-seo-sitemap-to-robots-txt
17140 Downloads
Add Yoast SEO Sitemap to robots.txt
hofff/contao-robots-txt-editor
123 Downloads
Editor for auto creation and modification of robots.txt.
text-media/robotstxt-parser
96 Downloads