Libraries tagged by craw
spatie/laravel-site-search
142990 Downloads
A site search engine
stil/curl-easy
215490 Downloads
cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot.
eliashaeussler/typo3-warming
307677 Downloads
Warming - Warms up Frontend caches based on an XML sitemap. Cache warmup can be triggered via TYPO3 backend or using a console command. Supports multiple languages and custom crawler implementations.
crwlr/crawler
19927 Downloads
Web crawling and scraping library.
crawlbase/crawlbase
69114 Downloads
A lightweight, dependency free PHP class that acts as wrapper for Crawlbase API
blackfire/player
17279 Downloads
A powerful web crawler and web scraper with Blackfire support
baba/sitemap-crawler
Downloads
smochin/instagram-php-crawler
10947 Downloads
A simple PHP Crawler for Instagram
nmure/crawler-detect-bundle
288643 Downloads
A Symfony bundle for the Crawler-Detect library (detects bots/crawlers/spiders via the user agent)
friends-of-hyva/magento2-crawler-session
8567 Downloads
Prevent crawlers from creating a session
aoepeople/crawler
290762 Downloads
Site Crawler - TYPO3 Crawler crawls the TYPO3 page tree. Used for cache warmup, indexing, publishing applications etc.
web-vision/ai-llms-txt
10787 Downloads
TYPO3 extension for generating .well-known/llm.txt links according to llmstxt.org specification
vipnytt/useragentparser
1042728 Downloads
User-Agent parser for robot rule sets
tomverran/robots-txt-checker
68255 Downloads
Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page
pushword/page-scanner
6596 Downloads
Find dead links, 404, 301 and more (command line or admin).