Libraries tagged by crawler

The Push API allows you to *push* items and security identities, as opposed to letting standard Coveo Cloud V2 crawlers *pull* this data from a content repository. This is especially useful when you need to index content from a cloud or on-premises system for which no dedicated source type exists in the Coveo Cloud V2 platform.

Go to Download

Generates an XML site map for use by web crawlers

Go to Download

Kirby robots.txt plugin for blocking AI Crawlers and Bots

Go to Download

Panth LLMs.txt — AI Indexing Engine for Magento 2. Serves structured /llms.txt, /llms-full.txt and /llms.json with a dynamic multi-URL sitemap engine, weighted ranking, auto-generated summaries, product-type and use-case grouping, and cron-based cache warm-up. Built for ChatGPT, Claude, Perplexity, Gemini and future LLM crawlers.

Go to Download

Ban AI crawlers from visiting your Symfony website or application

Go to Download

Lib para converter HTML em PDF de uma maneira segura para as necessidades dos crawlers.

Go to Download

Runs crawler to test site pages accessibility

Go to Download

Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫)，基于phpQuery。

Go to Download

Lightweight web crawler for php

Go to Download

A Laravel package to detect web crawlers via the user agent

Go to Download

A versatile and extendable crawler for TYPO3 CMS

Go to Download

Go to Download

Web crawler based on PHP Guzzle HTTP Client with concurrency support for faster operation.

Go to Download

Distributed Crawler Architecture

Go to Download

Redis CrawlQueue for spatie/crawler

Go to Download