Libraries tagged by site crawler

tomasnorre/crawler

58 Favers
477022 Downloads

Site Crawler - TYPO3 Crawler crawls the TYPO3 page tree. Used for cache warmup, indexing, publishing applications etc.

Go to Download


aoepeople/crawler

59 Favers
291164 Downloads

Site Crawler - TYPO3 Crawler crawls the TYPO3 page tree. Used for cache warmup, indexing, publishing applications etc.

Go to Download


schliesser/sitecrawler

11 Favers
41810 Downloads

TYPO3 sitemap crawler

Go to Download


jooservices/crawlerx

1 Favers
1470 Downloads

URL-driven crawl and parse library for JAV catalog sites.

Go to Download


bringyourownideas/laravel-sitemap

9 Favers
8149 Downloads

A simple website crawler & sitemap generator without a headless browser for Laravel 5.8+

Go to Download


sleeping-owl/apist

312 Favers
5329 Downloads

Package to provide api-like access to foreign sites based on html parsing

Go to Download


survos/crawler-bundle

1 Favers
6136 Downloads

Provides a way to create tests that crawl a site

Go to Download


vakata/downloader

0 Favers
1087 Downloads

A PHP class for making a static copy (or indexing) a site

Go to Download


pdir/contao-llms-bundle

0 Favers
72 Downloads

This bundle allows you to generate a llms.txt file via the Contao backend to optimize your site for LLMs like ChatGPT, Claude, Gemini, ...

Go to Download


omnisense/omnicrawl

0 Favers
59 Downloads

Omnicrawl: fetch and extract web pages under any PHP host, through crawling services or directly, responsible by default - the target site's rules applied by the family itself before any service is called. robots.txt read and honoured per RFC 9309 for the family's own token, a usage policy per target site declared as data (allowed or refused, on what authority, with a reference and a date, its rate limits, what may be stored and republished) and a conservative default for the others (single pages for previews, no crawl run), an identified User-Agent required by every factory, per-host limits through omnisense/contracts' PolicyGuard beside the provider's own, X-Robots-Tag, meta robots and TDMRep reservations honoured, provenance on every record. Three contracts: page-fetch, extract (metadata, Open Graph, JSON-LD, selectors), crawl-runs. A direct fetcher built in; omnicrawl/apify, omnicrawl/firecrawl, omnicrawl/scrapingbee, omnicrawl/zyte as packages of their own, none of their evasion features exposed.

Go to Download


glook/apist

0 Favers
571 Downloads

Package to provide api-like access to foreign sites based on html parsing (fork of sleeping-owl/apist package with updated dependencies)

Go to Download


anilp07x/scraper_vagas

1 Favers
390 Downloads

Motor de web scraping com estratégia leve (Guzzle + DomCrawler) ou pesada (Panther/Chrome headless) e adaptadores para sites de empresas em Angola.

Go to Download


bariew/yii2-doctest-extension

3 Favers
3304 Downloads

Runs crawler to test site pages accessibility

Go to Download


mjp91/sitemap-generator

1 Favers
19971 Downloads

Generates an XML site map for use by web crawlers

Go to Download


lastcall/last-call-crawler

4 Favers
61 Downloads

Static site crawler/processor

Go to Download


Next >>