Libraries tagged by The Web Robots Pages

t1gor/robots-txt-parser

90 Favers
500109 Downloads

PHP class to parse robots.txt rules according to Google, Yandex, W3C and The Web Robots Pages specifications.

27 Favers
775829 Downloads

Robots.txt parsing library, with full support for every directive and specification.