Libraries tagged by crawl

A fast, simple and powerful PHP web crawler (scraper/spider) 快速、简洁且强大的爬虫/采集框架

Go to Download

The Google Analytics module consists of 2 components that can be employed independently: The Google Logger injects the google analytics javascript snippet into your source code and logs relevant events (as of now only crawler visits) The Analyzer adds the Google Analytics UI to your CMS.

Go to Download

LLMs.txt Generator - Generates llms.txt files for AI/LLM crawlers with website content in Markdown format, with optional API key protection.

Go to Download

Crawl the web with kohana or phantomjs.

Go to Download

PHP Sitemap crawler

Go to Download

The Push API allows you to *push* items and security identities, as opposed to letting standard Coveo Cloud V2 crawlers *pull* this data from a content repository. This is especially useful when you need to index content from a cloud or on-premises system for which no dedicated source type exists in the Coveo Cloud V2 platform.

Go to Download

Kirby robots.txt plugin for blocking AI Crawlers and Bots

Go to Download

A custom middleware package for Laravel. Temporarily blocks crawlers scanning for vulnerabilities.

Go to Download

Strong link obfuscation with PHP and JavaScript. Object-oriented, using the Rot47 algorithm. Hides https, email, phone and sms links from being crawled by spambots.

Go to Download

Serve Markdown versions of your Craft CMS pages to AI crawlers and LLMs via .md URLs, content negotiation, and user-agent detection.

Go to Download

QueryList Plugin: Use Puppeteer to crawl Javascript dynamically rendered pages.(Headless Chrome ) 使用Puppeteer采集JavaScript动态渲染的页面

Go to Download

QueryList Plugin: Use PhantomJS to crawl Javascript dynamically rendered pages.(headless WebKit ) 使用PhantomJS采集JavaScript动态渲染的页面

Go to Download

A PHP library used to download, parse and crawl sitemap.xml files.

Go to Download

Utilities that are needed in multiple crawler packages.

Go to Download

Contao bundle helper class to detect search engines, bots, spiders, crawlers ...

Go to Download