Libraries tagged by zcrawler

Verify that web crawlers are actually who they claim to be.

Go to Download

Go to Download

Extract different parts of an HTML or XML document.

Go to Download

Yii2 extension for CrawlerDetect library

Go to Download

Shared, framework-agnostic building blocks for polite and safe web crawlers: SSRF-aware HTTP throttling, robots.txt parsing, and HTML link discovery.

Go to Download

A simple crawler (spider) writen in php just for fun

Go to Download

Lightweight proxy with full support for sessions, cookies, POST/FORM submissions, and URL rewriting.

Go to Download

PHP POO cURL wrapper (PSR friendly) : light and simple.

Go to Download

Tiny WebCrawler for Laravel using Playwright

Go to Download

Find dead links, 404, 301 and more (command line or admin).

Go to Download

A tool to crawl your own laravel installation checking your HTTP status codes

Go to Download

Integrates the Bit&Black Sitemap into a Kiwa website to create sitemap files seamlessly.

Go to Download

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性

Go to Download

A professional, production-ready PHP XML sitemap generator with advanced features: concurrency, robots.txt, caching, database storage, change detection, SEO analysis, JavaScript rendering, proxy support, webhooks, and more.

Go to Download

Bot and AI-crawler control presets for the Phirewall PHP firewall - block AI scrapers and rate-limit aggressive SEO crawlers

Go to Download