Libraries tagged by 64Robots
crwlr/robots-txt
8802 Downloads
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
bnomei/kirby3-robots-txt
37388 Downloads
Manage the robots.txt from the Kirby config file
vipnytt/useragentparser
756080 Downloads
User-Agent parser for robot rule sets
marcortola/behat-seo-contexts
93929 Downloads
Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...
glicer/link-checker
12381 Downloads
Check broken links in html files, sitemap.xml and robots.txt
cuab/phpcrawl
33288 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
monsieurbiz/sylius-robots-txt-plugin
546 Downloads
Manage your robot.txt from your Sylius admin pannel
merkandi/laravel-5-robots
9274 Downloads
Robots.txt Generator for Laravel 5
fifty-deg/sylius-robots-plugin
4196 Downloads
Channel defined robots.txt plugin for Sylius.
execut/yii2-robots-txt
11785 Downloads
Module for generate robots.txt file by url rules
evheniy/robots-txt-bundle
16367 Downloads
The RobotsTxtBundle adds the ability to generate robots.txt file to your application.
brittainmedia/phpcrawl
3453 Downloads
PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.
tidycode/meta-robots-tag
2264 Downloads
N/A
snicco/no-robots-middleware
3273 Downloads
razorcreations/nova-robots-editor
9211 Downloads
A Laravel Nova tool.