Libraries tagged by zcrawler

Bot and AI-crawler control presets for the Phirewall PHP firewall - block AI scrapers and rate-limit aggressive SEO crawlers

Go to Download

PHPScraper, built with simplicity in mind. See tests/ for more examples.

Go to Download

parser/handler for Robots Exclusion Protocol (robots.txt and more)

Go to Download

Marsvin it's a framework to write crawler applications

Go to Download

Water Melon is simple melon.com api sdk for php

Go to Download

A lightweight, extensible PHP library for scraping MMA data from multiple sources.

Go to Download

A web crawler, built to be easily dropped into PHP applications, which behaves just like a regular web browser, interpreting location redirects and storing cookies automatically.

Go to Download

Enhanced user agent parser for Laravel

Go to Download

Official, experimental (AI-generated and AI-maintained) Apify API client for PHP: run Actors, manage storages (datasets, key-value stores, request queues), schedules, webhooks and more.

Go to Download

Magento 2 module for AI Engine Optimization (AEO). Injects AI crawler rules (OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, ClaudeBot, anthropic-ai, Claude-User, Applebot, cohere-ai, Amazonbot, Meta-ExternalAgent) into robots.txt — without overwriting your existing configuration. Supports per-bot Allow/Disallow lists, Crawl-delay, Sitemap directives, multi-store, and a public Api\RobotsStatusInterface for cross-module integration with angeo/module-aeo-audit.

Go to Download

Laravel sitemap generator and crawler package for SEO optimization. Supports multilingual sites, image/video indexing, link validation, priority scoring, and advanced crawling automation.

Go to Download

Serve static content of a one-page website to crawlers (e.g. Facebook, Google or Twitter) using PhantomJS

Go to Download

Detect robots and crawlers.

Go to Download

A web scraper for PHP to easily extract data from web pages

Go to Download

PHP async scrapper used multi curl and reactphp, and proxy inspired by python grab

Go to Download