Libraries tagged by Martijn Koster
vipnytt/robotstxtparser
26 Favers
846514 Downloads
846514 Downloads
Robots.txt parsing library, with full support for every directive and specification.
Robots.txt parsing library, with full support for every directive and specification.