Libraries tagged by crawlora

A web scraper for PHP to easily extract data from web pages

Go to Download

Instant indexing and GEO for Craft CMS — push URLs to Google's Indexing API and IndexNow the moment they change, then manage, log and audit how AI search engines read your site.

Go to Download

A honeypot for bad bots — a hidden trap link that robots.txt tells crawlers to stay away from, and a permanent block for the ones that follow it anyway.

Go to Download

Protect your Laravel app from AI scrapers, LLM crawlers, and prompt injection attacks

Go to Download

PHP library providing functionality to verify that user-agents are who they claim to be.

Go to Download

Thumbsnag crawls an HTML document and finds imagery that best represents the given page.

Go to Download

Another GooglePlay Website Scraper based on goutte

Go to Download

A PHP desktop/mobile user agent parser with bot detection, based on Mobiledetect and CrawlerDetect.

Go to Download

parser/handler for Robots Exclusion Protocol (robots.txt and more)

Go to Download

Marsvin it's a framework to write crawler applications

Go to Download

Water Melon is simple melon.com api sdk for php

Go to Download

A web crawler, built to be easily dropped into PHP applications, which behaves just like a regular web browser, interpreting location redirects and storing cookies automatically.

Go to Download

PHPCreeper plugin for webman

Go to Download

Enhanced user agent parser for Laravel

Go to Download

Magento 2 module for AI Engine Optimization (AEO). Manages AI crawler rules in robots.txt (OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Google-Extended, ClaudeBot, Claude-SearchBot, Applebot-Extended, CCBot and more) without overwriting your existing configuration, emits IETF Content-Usage, Cloudflare Content-Signal and RSL License directives, and verifies that a crawler is genuine via Web Bot Auth request signatures (RFC 9421) or vendor-published IP ranges.

Go to Download