Libraries tagged by txt

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Go to Download

Manage the robots.txt from the Kirby config file

Go to Download

User-Agent parser for robot rule sets

Go to Download

CLI tool for converting WordPress Plugin readme.txt to Markdown

Go to Download

Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...

Go to Download

Export HTML table to CSV, DOC, JSON, PDF, PNG, SQL, TSV, TXT, XLS, XLSX, XML

Go to Download

Check broken links in html files, sitemap.xml and robots.txt

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

Automates syncing of changelog entries from CHANGELOG.md to WordPress plugin readme.txt.

Go to Download

Module for generate robots.txt file by url rules

Go to Download

The RobotsTxtBundle adds the ability to generate robots.txt file to your application.

Go to Download

PHP SDK for ReportingCloud Web API. Authored and supported by Text Control GmbH.

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

Robots.txt Generator for Laravel 5

Go to Download

Kirby robots.txt plugin for blocking AI Crawlers and Bots

Go to Download