Libraries tagged by zcrawler

The Push API allows you to *push* items and security identities, as opposed to letting standard Coveo Cloud V2 crawlers *pull* this data from a content repository. This is especially useful when you need to index content from a cloud or on-premises system for which no dedicated source type exists in the Coveo Cloud V2 platform.

Go to Download

Kirby robots.txt plugin for blocking AI Crawlers and Bots

Go to Download

A custom middleware package for Laravel. Temporarily blocks crawlers scanning for vulnerabilities.

Go to Download

Obfuscate emails, phone numbers and other sensitive data using PHP and modern web features. Visible to humans, hidden from crawlers and headless bots until they interact.

Go to Download

Magento2: prevent session creation for bots & crawlers.

Go to Download

Contao bundle helper class to detect search engines, bots, spiders, crawlers ...

Go to Download

A crawler to find all unique internal pages on a given website

Go to Download

宇润爬虫框架(Yurun Crawler) 是一个低代码、高性能、分布式爬虫采集框架，这可能是最一把梭的爬虫框架。

Go to Download

Crawls Website and can generate backstop json

Go to Download

Boost Lighthouse and PageSpeed scores by hiding cookie banners, chat widgets and heavy scripts from crawlers via two Blade directives. Octane-safe.

Go to Download

Wrapper for the crawler detection package

Go to Download

Crawls all unique links.

Go to Download

Provides a way to create tests that crawl a site

Go to Download

Web Crawler to check few SEO basics.

Go to Download

An full search page crawler to enable complex and customized searching abilities.

Go to Download