Libraries tagged by crawl

A custom middleware package for Laravel. Temporarily blocks crawlers scanning for vulnerabilities.

Go to Download

Strong link obfuscation with PHP and JavaScript. Object-oriented, using the Rot47 algorithm. Hides https, email, phone and sms links from being crawled by spambots.

Go to Download

Serve Markdown versions of your Craft CMS pages to AI crawlers and LLMs via .md URLs, content negotiation, and user-agent detection.

Go to Download

QueryList Plugin: Use Puppeteer to crawl Javascript dynamically rendered pages.(Headless Chrome ) 使用Puppeteer采集JavaScript动态渲染的页面

Go to Download

QueryList Plugin: Use PhantomJS to crawl Javascript dynamically rendered pages.(headless WebKit ) 使用PhantomJS采集JavaScript动态渲染的页面

Go to Download

A PHP library used to download, parse and crawl sitemap.xml files.

Go to Download

Utilities that are needed in multiple crawler packages.

Go to Download

Contao bundle helper class to detect search engines, bots, spiders, crawlers ...

Go to Download

A Magento 2 module that improves store performance by automatically warming cache through intelligent page crawling and async refresh mechanisms.

Go to Download

宇润爬虫框架(Yurun Crawler) 是一个低代码、高性能、分布式爬虫采集框架，这可能是最一把梭的爬虫框架。

Go to Download

【正方教务】爬虫，支持成绩查询、考试查询、课表查询、四六级成绩查询、四六级报名、选课查询、修改密码、获取用户菜单等功能，并且解析数据成易读格式，符合 psr 规范，拿来即用

Go to Download

Crawls Website and can generate backstop json

Go to Download

Boost Lighthouse and PageSpeed scores by hiding cookie banners, chat widgets and heavy scripts from crawlers via two Blade directives. Octane-safe.

Go to Download

Crawls all unique links.

Go to Download

Provides a way to create tests that crawl a site

Go to Download