Libraries tagged by robots-txt

Determine if a page may be crawled from robots.txt and robots meta tags

Go to Download

XML Sitemap parser class compliant with the Sitemaps.org protocol.

Go to Download

Models a robots.txt file

Go to Download

PHP class to parse robots.txt rules according to Google, Yandex, W3C and The Web Robots Pages specifications.

Go to Download

PHP Class for parsing robots.txt files according to Google, Yandex specifications.

Go to Download

Robots.txt parsing library, with full support for every directive and specification.

Go to Download

Set the robots.txt content dynamically based on the Laravel app environment.

Go to Download

Novactive eZ SEO Bundle is an Ibexa Platform bundle for SEO simplications. metas, sitemaps, robots.txt, etc.

Go to Download

Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...

Go to Download

Robots.txt Generator for Laravel 4 and 5

Go to Download

Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page

Go to Download

Generate robots.txt

Go to Download

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Go to Download

Manage a virtual robots.txt from the Kirby config file

Go to Download

User-Agent parser for robot rule sets

Go to Download