Libraries tagged by crawler
gpupo/camelspider
1 Downloads
A class to help handle gateways to processing entities.
danack/sitetool
33 Downloads
Event based site tool and tester.
core23/lastfm-api
2815 Downloads
Last.fm webservice client for php.
appstract/laravel-crawl-render
37 Downloads
Pre-render your site for web crawlers
webdecoy/sensor
4 Downloads
Report bot and AI-crawler detections to WebDecoy from any PHP application
justinholtweb/craft-sanka
24 Downloads
Instant indexing and GEO for Craft CMS — push URLs to Google's Indexing API and IndexNow the moment they change, then manage, log and audit how AI search engines read your site.
fastwhale/phpscraper
42 Downloads
PHPScraper, built with simplicity in mind. See tests/ for more examples.
bleuren/agent
105 Downloads
Enhanced user agent parser for Laravel
angeo/module-robots-txt-aeo
18 Downloads
Magento 2 module for AI Engine Optimization (AEO). Manages AI crawler rules in robots.txt (OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Google-Extended, ClaudeBot, Claude-SearchBot, Applebot-Extended, CCBot and more) without overwriting your existing configuration, emits IETF Content-Usage, Cloudflare Content-Signal and RSL License directives, and verifies that a crawler is genuine via Web Bot Auth request signatures (RFC 9421) or vendor-published IP ranges.
zeix/craft-marked-down
15 Downloads
Marked Down serves Markdown to AI crawlers, command-line tools, and APIs while browsers get HTML. Zero-config per-section setup, automatic AI bot detection, .md URL suffix, /llms.txt, YAML front matter, and dedicated Markdown templates — out of the box.
yggverse/yo
10 Downloads
Yo! Micro Web Crawler in PHP & Manticore
ssola/crawly
18 Downloads
Simple web crawler library
phpscrapers/netflix
19 Downloads
Unofficial API crawler to get Netflix viewing history.
peteryang/querylist
289 Downloads
Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫),基于phpQuery。
orangecat/feed
3 Downloads
Generate product feed for AI Crawlers