PHP code example of innmind / robots-txt

1. Go to this page and download the library: Download innmind/robots-txt library. Choose the download type require.

2. Extract the ZIP file and open the index.php.

3. Add this code to the index.php.

    
        
<?php
require_once('vendor/autoload.php');

/* Start to develop here. Best regards https://php-download.com/ */

innmind / robots-txt example snippets


use Innmind\RobotsTxt\Parser;
use Innmind\OperatingSystem\Factory;
use Innmind\Url\Url;

$os = Factory::build();
$parse = Parser::of(
    $os->remote()->http(),
    'My user agent',
);
$robots = $parse(Url::of('https://github.com/robots.txt'))->match(
    static fn($robots) => $robots,
    static fn() => throw new \RuntimeException('robots.txt not found'),
);
$robots->disallows('My user agent', Url::of('/humans.txt')); //false
$robots->disallows('My user agent', Url::of('/any/other/url')); //true