Libraries tagged by robots-txt
spatie/robots-txt
23303100 Downloads
Determine if a page may be crawled from robots.txt and robots meta tags
vipnytt/sitemapparser
2612463 Downloads
XML Sitemap parser class compliant with the Sitemaps.org protocol.
webignition/robots-txt-file
1594346 Downloads
Models a robots.txt file
verschuur/laravel-robotstxt
491514 Downloads
Set the robots.txt content dynamically based on the Laravel app environment.
bopoda/robots-txt-parser
338363 Downloads
PHP Class for parsing robots.txt files according to Google, Yandex specifications.
vipnytt/robotstxtparser
849986 Downloads
Robots.txt parsing library, with full support for every directive and specification.
t1gor/robots-txt-parser
512647 Downloads
PHP class to parse robots.txt rules according to Google, Yandex, W3C and The Web Robots Pages specifications.
novactive/ezseobundle
271879 Downloads
Novactive eZ SEO Bundle is an Ibexa Platform bundle for SEO simplications. metas, sitemaps, robots.txt, etc.
healey/robots
214681 Downloads
Robots.txt Generator for Laravel 4 and 5
tomverran/robots-txt-checker
69558 Downloads
Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page
crwlr/robots-txt
41705 Downloads
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
frosh/robots-txt
39558 Downloads
Generate robots.txt
bnomei/kirby3-robots-txt
51860 Downloads
Manage a virtual robots.txt from the Kirby config file
vipnytt/useragentparser
1058475 Downloads
User-Agent parser for robot rule sets
bertugfahriozer/ci4seopro
6094 Downloads
Professional SEO/AI indexing + Sitemap (index) + RSS + News/Image/Video sitemaps for CodeIgniter 4