Libraries tagged by robots-exclusion-protocol

vipnytt/robotstxtparser

25 Favers
556928 Downloads

Robots.txt parsing library, with full support for every directive and specification.

19 Favers
33243 Downloads

Manage the robots.txt from the Kirby config file

2 Favers
711051 Downloads

User-Agent parser for robot rule sets

7 Favers
184 Downloads

parser/handler for Robots Exclusion Protocol (robots.txt and more)