Libraries tagged by crawler

gpupo/camelspider

11 Favers
1 Downloads

A class to help handle gateways to processing entities.

Go to Download


danack/sitetool

5 Favers
33 Downloads

Event based site tool and tester.

Go to Download


core23/lastfm-api

18 Favers
2815 Downloads

Last.fm webservice client for php.

Go to Download


appstract/laravel-crawl-render

11 Favers
37 Downloads

Pre-render your site for web crawlers

Go to Download


webdecoy/sensor

0 Favers
4 Downloads

Report bot and AI-crawler detections to WebDecoy from any PHP application

Go to Download


justinholtweb/craft-sanka

1 Favers
24 Downloads

Instant indexing and GEO for Craft CMS — push URLs to Google's Indexing API and IndexNow the moment they change, then manage, log and audit how AI search engines read your site.

Go to Download


fastwhale/phpscraper

0 Favers
42 Downloads

PHPScraper, built with simplicity in mind. See tests/ for more examples.

Go to Download


bleuren/agent

0 Favers
105 Downloads

Enhanced user agent parser for Laravel

Go to Download


angeo/module-robots-txt-aeo

5 Favers
18 Downloads

Magento 2 module for AI Engine Optimization (AEO). Manages AI crawler rules in robots.txt (OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Google-Extended, ClaudeBot, Claude-SearchBot, Applebot-Extended, CCBot and more) without overwriting your existing configuration, emits IETF Content-Usage, Cloudflare Content-Signal and RSL License directives, and verifies that a crawler is genuine via Web Bot Auth request signatures (RFC 9421) or vendor-published IP ranges.

Go to Download


zeix/craft-marked-down

4 Favers
15 Downloads

Marked Down serves Markdown to AI crawlers, command-line tools, and APIs while browsers get HTML. Zero-config per-section setup, automatic AI bot detection, .md URL suffix, /llms.txt, YAML front matter, and dedicated Markdown templates — out of the box.

Go to Download


yggverse/yo

4 Favers
10 Downloads

Yo! Micro Web Crawler in PHP & Manticore

Go to Download


ssola/crawly

4 Favers
18 Downloads

Simple web crawler library

Go to Download


phpscrapers/netflix

6 Favers
19 Downloads

Unofficial API crawler to get Netflix viewing history.

Go to Download


peteryang/querylist

0 Favers
289 Downloads

Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫),基于phpQuery。

Go to Download


orangecat/feed

4 Favers
3 Downloads

Generate product feed for AI Crawlers

Go to Download


<< Previous Next >>