Download the PHP package ranvis/robots-txt-processor without Composer

On this page you can find all versions of the php package ranvis/robots-txt-processor. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.

FAQ

After the download, you have to make one include require_once('vendor/autoload.php');. After that you have to import the classes with use statements.

Example:
If you use only one package a project is not needed. But if you use more then one package, without a project it is not possible to import the classes with use statements.

In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
Some PHP packages are not free to download and because of that hosted in private repositories. In this case some credentials are needed to access such packages. Please use the auth.json textarea to insert credentials, if a package is coming from a private repository. You can look here for more information.

  • Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
  • To use Composer is sometimes complicated. Especially for beginners.
  • Composer needs much resources. Sometimes they are not available on a simple webspace.
  • If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
  • Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Please rate this library. Is it a good library?

Informations about the package robots-txt-processor

robots.txt filter and tester for untrusted source

Introduction

robots-txt-processor is a tester with a filter for natural wild robots.txt data of the Internet. The module can filter like:

Also, it can for example:

Tester module can process Allow/Disallow directives containing */$ meta characters. Alternatively, you can use the filter module alone and feed an output to another tester module as a single User-agent: * record with a non-group record (e.g. Sitemap.)

License

BSD 2-Clause License

Installation

composer require "ranvis/robots-txt-processor:^1.0"

Example Usage

Tester->setSource(string) is actually a shorthand of Tester->setSource(RecordSet):

See EXAMPLES.md for more examples, including filter-only usage.

Implementation Notes

Setting user-agents

When setting source, you can (optionally) pass user-agents like the examples above. If you pass a user-agent string or an array of strings, subsequent Filter will filter out unspecified user-agent records (aside from *.) While Tester->isAllowed() accepts user-agents, it should run faster to filter (with Filter->setUserAgents() or Tester->setSource(source, userAgents)) and call Tester->isAllowed() multiple times without specifying user-agents. (When an array of user-agent strings is passed, a user-agent specified earlier takes precedence when testing.)

Record separator

This parser ignores blank lines. Another record starts on User-agent lines after group member lines (i.e. Disallow/Allow.)

Case sensitivity

User-agent value and directive names like Disallow are case-insensitive. Filter class normalizes directive names to First-character-uppercased form.

Encoding conversion

This filter/tester themselves don't handle encoding conversion because it isn't needed. If a remote robots.txt uses some non-Unicode (specifically not UTF-8) encoding, URL path should be in that encoding too. The filter/tester safely work with any character or percent-encoded sequence which can result in invalid UTF-8. An exception is when a remote robots.txt uses any Unicode encoding with BOM. If this will ever happen, you will need to convert it to UTF-8 (without BOM) beforehand.

Features

See features/behaviors table of robots-txt-processor-test project.

Options

Options can be specified in the first argument of constructors. Normally, the default values should suffice to filter potentially offensive input while preserving requested rules.

Tester class options

Tester->setSource(string) internally instantiates Filter with initially passed options and calls Filter->getRecordSet(string).

Filter class options

Filter->getRecordSet(string) internally instantiates FilterParser with initially passed options.

FilterParser class options

FilterParser extends Parser class.

Parser class options

Interface


All versions of robots-txt-processor with dependencies

PHP Build Version
Package Version
Requires php Version >=7.3.0
Composer command for our command line client (download client) This client runs in each environment. You don't need a specific PHP version etc. The first 20 API calls are free. Standard composer command

The package ranvis/robots-txt-processor contains the following files

Loading the files please wait ....