Libraries tagged by robots

Simple robots generation module for Silverstripe

Go to Download

Laravel package to manage robots in an easy way

Go to Download

Robots.txt generator service.

Go to Download

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Go to Download

钉钉智能群助手 Laravel/Lumen 消息通知扩展包（Dingtalk robot message notifications for Laravel/Lumen）

Go to Download

Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page

Go to Download

Generate robots.txt

Go to Download

User-Agent parser for robot rule sets

Go to Download

Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...

Go to Download

Automated tests for the Robot Framework

Go to Download

Check broken links in html files, sitemap.xml and robots.txt

Go to Download

Indivdual indexing/robot information for each news article record

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

钉钉、企业微信、飞书 自定义群组机器人。支持链式语法创建发送消息。支持文本、Markdown、图片、文件、图文、卡片等消息。

Go to Download

Symfony2 bundle to control X-Robots-Tag HTTP header via annotations

Go to Download