Libraries tagged by Robots Exclusion Protocol

vipnytt/robotstxtparser

27 Favers
708418 Downloads

Robots.txt parsing library, with full support for every directive and specification.

21 Favers
45874 Downloads

Manage a virtual robots.txt from the Kirby config file

12 Favers
18113 Downloads

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

2 Favers
887663 Downloads

User-Agent parser for robot rule sets

7 Favers
187 Downloads

parser/handler for Robots Exclusion Protocol (robots.txt and more)