Libraries tagged by tx

Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page

Go to Download

Generate robots.txt

Go to Download

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Go to Download

Manage the robots.txt from the Kirby config file

Go to Download

PHP SDK for ReportingCloud Web API. Authored and supported by Text Control GmbH.

Go to Download

User-Agent parser for robot rule sets

Go to Download

A PHP library to render an array representing a matrix as a plain text table.

Go to Download

CLI tool for converting WordPress Plugin readme.txt to Markdown

Go to Download

Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...

Go to Download

Robots.txt generator service.

Go to Download

Export HTML table to CSV, DOC, JSON, PDF, PNG, SQL, TSV, TXT, XLS, XLSX, XML

Go to Download

Check broken links in html files, sitemap.xml and robots.txt

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

Module for generate robots.txt file by url rules

Go to Download

The RobotsTxtBundle adds the ability to generate robots.txt file to your application.

Go to Download