Libraries tagged by robots.txt

spatie/robots-txt

213 Favers
7720537 Downloads

Determine if a page may be crawled from robots.txt and robots meta tags

webignition/robots-txt-file

19 Favers
790644 Downloads

Models a robots.txt file

t1gor/robots-txt-parser

87 Favers
399401 Downloads

PHP class to parse robots.txt rules according to Google, Yandex, W3C and The Web Robots Pages specifications.

vipnytt/sitemapparser

75 Favers
1171588 Downloads

XML Sitemap parser class compliant with the Sitemaps.org protocol.

vipnytt/robotstxtparser

25 Favers
547418 Downloads

Robots.txt parsing library, with full support for every directive and specification.

verschuur/laravel-robotstxt

30 Favers
217459 Downloads

Set the robots.txt content dynamically based on the Laravel app environment.

bopoda/robots-txt-parser

42 Favers
142484 Downloads

PHP Class for parsing robots.txt files according to Google, Yandex specifications.

novactive/ezseobundle

28 Favers
200725 Downloads

Novactive eZ SEO Bundle is an Ibexa Platform bundle for SEO simplications. metas, sitemaps, robots.txt, etc.

healey/robots

40 Favers
167948 Downloads

Robots.txt Generator for Laravel 4 and 5

crwlr/robots-txt

8 Favers
5157 Downloads

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

frosh/robots-txt

9 Favers
5406 Downloads

Generate robots.txt

bnomei/kirby3-robots-txt

19 Favers
31151 Downloads

Manage the robots.txt from the Kirby config file

vipnytt/useragentparser

2 Favers
700468 Downloads

User-Agent parser for robot rule sets

marcortola/behat-seo-contexts

12 Favers
87156 Downloads

Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...

mad-web/laravel-robots

33 Favers
126553 Downloads

Robots.txt generator service.

Next >>