Download the PHP package vipnytt/robotstxtparser without Composer
On this page you can find all versions of the php package vipnytt/robotstxtparser. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Download vipnytt/robotstxtparser
More information about vipnytt/robotstxtparser
Files in vipnytt/robotstxtparser
Package robotstxtparser
Short Description Robots.txt parsing library, with full support for every directive and specification.
License MIT
Homepage https://github.com/VIPnytt/RobotsTxtParser
Informations about the package robotstxtparser
Robots.txt parser
An easy to use, extensible robots.txt
parser library with full support for literally every specification on the Internet.
Usage cases:
- Permission checks
- Fetch crawler rules
- Sitemap discovery
- Host preference
- Dynamic URL parameter discovery
robots.txt
rendering
Advantages
(compared to most other robots.txt libraries)
- Automatic
robots.txt
download. (optional) - Integrated Caching system. (optional)
- Crawl Delay handler.
- Documentation available.
- Support for literally every specification.
- HTTP Status code handler, _according to Google's spec._
- Dedicated
User-Agent
parser and group determiner library, for maximum accuracy. - Provides additional data like preferred host, dynamic URL parameters, Sitemap locations, etc.
- Protocols supported:
HTTP
,HTTPS
,FTP
,SFTP
andFTP/S
.
Requirements:
Installation
The recommended way to install the robots.txt parser is through Composer. Add this to your composer.json
file:
Then run:
Getting started
Basic usage example
A small excerpt of basic methods
The above is just a taste the basics, a whole bunch of more advanced and/or specialized methods are available for almost any purpose. Visit the cheat-sheet for the technical details.
Visit the Documentation for more information.
Directives
Specifications
- Google robots.txt specifications
- Yandex robots.txt specifications
- W3C Recommendation HTML 4.01 specification
- Sitemaps.org protocol
- Sean Conner: "An Extended Standard for Robot Exclusion"
- Martijn Koster: "A Method for Web Robots Control"
- Martijn Koster: "A Standard for Robot Exclusion"
- RFC 7231,
2616 - RFC 7230,
2616 - RFC 5322,
2822,822 - RFC 3986,
1808 - RFC 1945
- RFC 1738
- RFC 952
All versions of robotstxtparser with dependencies
ext-curl Version *
ext-mbstring Version *
composer/ca-bundle Version ^1.0
vipnytt/useragentparser Version ^1.0