Libraries tagged by crawler

Obfuscate emails, phone numbers and other sensitive data using PHP and modern web features. Visible to humans, hidden from crawlers and headless bots until they interact.

Go to Download

Magento2: prevent session creation for bots & crawlers.

Go to Download

Utilities that are needed in multiple crawler packages.

Go to Download

Contao bundle helper class to detect search engines, bots, spiders, crawlers ...

Go to Download

A crawler to find all unique internal pages on a given website

Go to Download

宇润爬虫框架(Yurun Crawler) 是一个低代码、高性能、分布式爬虫采集框架，这可能是最一把梭的爬虫框架。

Go to Download

【正方教务】爬虫，支持成绩查询、考试查询、课表查询、四六级成绩查询、四六级报名、选课查询、修改密码、获取用户菜单等功能，并且解析数据成易读格式，符合 psr 规范，拿来即用

Go to Download

Crawls Website and can generate backstop json

Go to Download

Provides a way to create tests that crawl a site

Go to Download

Web Crawler to check few SEO basics.

Go to Download

An full search page crawler to enable complex and customized searching abilities.

Go to Download

Modern library to handle RSS/Atom feeds

Go to Download

Standalone Laravel package for queued Flickr crawling with rate limiting and MySQL persistence.

Go to Download

Extension for the crwlr/crawler package containing steps utilizing a headless browser.

Go to Download

Crawl sitemap.xml recursively and return collected urls as array

Go to Download