Libraries tagged by site crawler
tomasnorre/crawler
477022 Downloads
Site Crawler - TYPO3 Crawler crawls the TYPO3 page tree. Used for cache warmup, indexing, publishing applications etc.
aoepeople/crawler
291164 Downloads
Site Crawler - TYPO3 Crawler crawls the TYPO3 page tree. Used for cache warmup, indexing, publishing applications etc.
schliesser/sitecrawler
41810 Downloads
TYPO3 sitemap crawler
jooservices/crawlerx
1470 Downloads
URL-driven crawl and parse library for JAV catalog sites.
bringyourownideas/laravel-sitemap
8149 Downloads
A simple website crawler & sitemap generator without a headless browser for Laravel 5.8+
sleeping-owl/apist
5329 Downloads
Package to provide api-like access to foreign sites based on html parsing
survos/crawler-bundle
6136 Downloads
Provides a way to create tests that crawl a site
vakata/downloader
1087 Downloads
A PHP class for making a static copy (or indexing) a site
pdir/contao-llms-bundle
72 Downloads
This bundle allows you to generate a llms.txt file via the Contao backend to optimize your site for LLMs like ChatGPT, Claude, Gemini, ...
omnisense/omnicrawl
59 Downloads
Omnicrawl: fetch and extract web pages under any PHP host, through crawling services or directly, responsible by default - the target site's rules applied by the family itself before any service is called. robots.txt read and honoured per RFC 9309 for the family's own token, a usage policy per target site declared as data (allowed or refused, on what authority, with a reference and a date, its rate limits, what may be stored and republished) and a conservative default for the others (single pages for previews, no crawl run), an identified User-Agent required by every factory, per-host limits through omnisense/contracts' PolicyGuard beside the provider's own, X-Robots-Tag, meta robots and TDMRep reservations honoured, provenance on every record. Three contracts: page-fetch, extract (metadata, Open Graph, JSON-LD, selectors), crawl-runs. A direct fetcher built in; omnicrawl/apify, omnicrawl/firecrawl, omnicrawl/scrapingbee, omnicrawl/zyte as packages of their own, none of their evasion features exposed.
glook/apist
571 Downloads
Package to provide api-like access to foreign sites based on html parsing (fork of sleeping-owl/apist package with updated dependencies)
anilp07x/scraper_vagas
390 Downloads
Motor de web scraping com estratégia leve (Guzzle + DomCrawler) ou pesada (Panther/Chrome headless) e adaptadores para sites de empresas em Angola.
bariew/yii2-doctest-extension
3304 Downloads
Runs crawler to test site pages accessibility
mjp91/sitemap-generator
19971 Downloads
Generates an XML site map for use by web crawlers
lastcall/last-call-crawler
61 Downloads
Static site crawler/processor