Libraries tagged by web crawler

Shared, framework-agnostic building blocks for polite and safe web crawlers: SSRF-aware HTTP throttling, robots.txt parsing, and HTML link discovery.

Go to Download

A professional, production-ready PHP XML sitemap generator with advanced features: concurrency, robots.txt, caching, database storage, change detection, SEO analysis, JavaScript rendering, proxy support, webhooks, and more.

Go to Download

A modern, fluent PHP package for managing robots.txt rules with type safety and great developer experience

Go to Download

Tiny WebCrawler for Laravel using Playwright

Go to Download

Lightweight proxy with full support for sessions, cookies, POST/FORM submissions, and URL rewriting.

Go to Download

Protect your Laravel app from AI scrapers, LLM crawlers, and prompt injection attacks

Go to Download

Crawlzone is a fast asynchronous internet crawling framework aiming to provide open source web search and testing solution. It can be used for a wide range of purposes, from extracting and indexing structured data to monitoring and automated testing.

Go to Download

Magento 2 AEO (AI Engine Optimization) Audit. v4 covers 18 signals across two layers: configuration checks (robots.txt with purpose-classified AI bots, llms.txt + llms.jsonl, Product / Organization / FAQ schema, merchant return + shipping policies, sitemap.xml, UCP profile, AI product feed, OG tags, canonical + hreflang, JSON-LD quality, well-known endpoint matrix, Core Web Vitals via CrUX) and an evidence layer (WAF-vs-robots.txt reality check, observed AI crawler activity via GDPR-safe built-in instrumentation or access-log source). Score Trend dashboard, Admin UI, configurable cron, CI-ready CLI, dependency-injected extension points for custom checkers and evidence sources.

Go to Download

A fast, simple and powerful PHP web crawler (scraper/spider) 快速、简洁且强大的爬虫/采集框架

Go to Download

Verify that web crawlers are actually who they claim to be.

Go to Download

Generic Scrapy-like web crawler for Laravel — auto-detects lists, pagination, fields, and supports API/SPA, infinite scroll, image downloading, dedup, resumable crawls, and pluggable output sinks.

Go to Download

Boost Lighthouse and PageSpeed scores by hiding cookie banners, chat widgets and heavy scripts from crawlers via two Blade directives. Octane-safe.

Go to Download

A PHP SDK for WebCrawler API - turn website into data

Go to Download

Motor de web scraping com estratégia leve (Guzzle + DomCrawler) ou pesada (Panther/Chrome headless) e adaptadores para sites de empresas em Angola.

Go to Download

Generates an XML site map for use by web crawlers

Go to Download