Libraries tagged by robots-exclusion-protocol

vipnytt/robotstxtparser

26 Favers
824018 Downloads

Robots.txt parsing library, with full support for every directive and specification.

2 Favers
1025275 Downloads

User-Agent parser for robot rule sets

22 Favers
50851 Downloads

Manage a virtual robots.txt from the Kirby config file

7 Favers
187 Downloads

parser/handler for Robots Exclusion Protocol (robots.txt and more)