Libraries tagged by robotstxt

spatie/robots-txt

225 Favers
9628414 Downloads

Determine if a page may be crawled from robots.txt and robots meta tags

vipnytt/sitemapparser

78 Favers
1454536 Downloads

XML Sitemap parser class compliant with the Sitemaps.org protocol.

webignition/robots-txt-file

19 Favers
923907 Downloads

Models a robots.txt file

vipnytt/robotstxtparser

27 Favers
614590 Downloads

Robots.txt parsing library, with full support for every directive and specification.

t1gor/robots-txt-parser

88 Favers
464051 Downloads

PHP class to parse robots.txt rules according to Google, Yandex, W3C and The Web Robots Pages specifications.

verschuur/laravel-robotstxt

37 Favers
268437 Downloads

Set the robots.txt content dynamically based on the Laravel app environment.

bopoda/robots-txt-parser

45 Favers
167132 Downloads

PHP Class for parsing robots.txt files according to Google, Yandex specifications.

novactive/ezseobundle

28 Favers
220322 Downloads

Novactive eZ SEO Bundle is an Ibexa Platform bundle for SEO simplications. metas, sitemaps, robots.txt, etc.

healey/robots

40 Favers
181251 Downloads

Robots.txt Generator for Laravel 4 and 5

tomverran/robots-txt-checker

13 Favers
39343 Downloads

Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page

crwlr/robots-txt

10 Favers
10440 Downloads

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

frosh/robots-txt

9 Favers
11690 Downloads

Generate robots.txt

bnomei/kirby3-robots-txt

20 Favers
39406 Downloads

Manage the robots.txt from the Kirby config file

vipnytt/useragentparser

2 Favers
777172 Downloads

User-Agent parser for robot rule sets

marcortola/behat-seo-contexts

13 Favers
96578 Downloads

Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...

Next >>