Libraries tagged by scrawl
aoepeople/crawler
290613 Downloads
Site Crawler - TYPO3 Crawler crawls the TYPO3 page tree. Used for cache warmup, indexing, publishing applications etc.
zrashwani/arachnid
20414 Downloads
A crawler to find all unique internal pages on a given website
web-vision/ai-llms-txt
9487 Downloads
TYPO3 extension for generating .well-known/llm.txt links according to llmstxt.org specification
vipnytt/useragentparser
1029627 Downloads
User-Agent parser for robot rule sets
tomverran/robots-txt-checker
67359 Downloads
Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page
opensearchserver/opensearchserver
68532 Downloads
PHP library for OpenSearchServer: professionnal search engine, crawlers (web, file, database), REST APIs, .... This library uses OpenSearchServer's V2 API.
nucleos/lastfm
13984 Downloads
Last.fm webservice client for php.
helgesverre/firecrawl-php-sdk
15623 Downloads
PHP SDK for Firecrawl API - Web scraping and crawling service
fkrzski/robots-txt
3063 Downloads
A modern, fluent PHP package for managing robots.txt rules with type safety and great developer experience
firecrawl/firecrawl-sdk
14374 Downloads
PHP SDK for the Firecrawl v2 API with Laravel support
eddieace/php-simple
66561 Downloads
crwlr/robots-txt
38795 Downloads
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
spatie/laravel-link-checker
53448 Downloads
Check all links in a Laravel app
spatie/http-status-check
48538 Downloads
CLI tool to crawl a website and check HTTP status code
proxycrawl/proxycrawl
110416 Downloads
A lightweight, dependency free PHP class that acts as wrapper for ProxyCrawl API