Libraries tagged by robotstxt
spatie/robots-txt
8336164 Downloads
Determine if a page may be crawled from robots.txt and robots meta tags
webignition/robots-txt-file
837391 Downloads
Models a robots.txt file
t1gor/robots-txt-parser
423589 Downloads
PHP class to parse robots.txt rules according to Google, Yandex, W3C and The Web Robots Pages specifications.
vipnytt/sitemapparser
1264710 Downloads
XML Sitemap parser class compliant with the Sitemaps.org protocol.
vipnytt/robotstxtparser
569892 Downloads
Robots.txt parsing library, with full support for every directive and specification.
verschuur/laravel-robotstxt
233486 Downloads
Set the robots.txt content dynamically based on the Laravel app environment.
bopoda/robots-txt-parser
150196 Downloads
PHP Class for parsing robots.txt files according to Google, Yandex specifications.
novactive/ezseobundle
208387 Downloads
Novactive eZ SEO Bundle is an Ibexa Platform bundle for SEO simplications. metas, sitemaps, robots.txt, etc.
healey/robots
172549 Downloads
Robots.txt Generator for Laravel 4 and 5
tomverran/robots-txt-checker
32071 Downloads
Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page
crwlr/robots-txt
6839 Downloads
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
frosh/robots-txt
7036 Downloads
Generate robots.txt
bnomei/kirby3-robots-txt
34765 Downloads
Manage the robots.txt from the Kirby config file
vipnytt/useragentparser
726638 Downloads
User-Agent parser for robot rule sets
marcortola/behat-seo-contexts
90936 Downloads
Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...