Libraries tagged by Robots Exclusion Protocol

vipnytt/robotstxtparser

27 Favers
615037 Downloads

Robots.txt parsing library, with full support for every directive and specification.

20 Favers
39438 Downloads

Manage the robots.txt from the Kirby config file

10 Favers
10459 Downloads

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

2 Favers
777702 Downloads

User-Agent parser for robot rule sets

7 Favers
185 Downloads

parser/handler for Robots Exclusion Protocol (robots.txt and more)