Download the PHP package commently/crawler without Composer

Laravel Polite Crawler

A minimal, polite URL crawler for Laravel. Give it a URL and it brings you the raw document — respecting robots.txt, per-host rate limits and conditional requests. It does not know (or care) what it crawls. Parsing and consuming the content is entirely your job, so the package plugs into any project without dictating a schema.

Why this package

Most "crawler" packages are opinionated about what you crawl: they discover links, follow HTML, render JavaScript, or ship their own database schema. laravel-polite-crawler deliberately does none of that. It is just the polite transport layer:

• what it does — fetch a URL (or a batch of URLs) over HTTP, politely and concurrently, and hand you the raw body plus transport metadata;
• what it doesn't do — parse feeds, scrape HTML, store rows, decide when to fetch again.

The collection pipeline stays in your application:

Features

• Polite by default — a single honest User-Agent, no browser impersonation.
• robots.txt — consulted (and cached) per host; a failure to fetch it never blocks.
• Per-host rate limiting — minimum delay between two requests to the same host and at most N concurrent requests per host, enforced across processes via the distributed cache.
• Global concurrency cap — simultaneous requests across all hosts, batched in waves.
• Conditional requests — sends If-None-Match / If-Modified-Since and reports 304 Not Modified instead of re-downloading.
• Outcome classification — every attempt ends in one of five explicit outcomes (success, not_modified, error, throttled, blocked), with Retry-After parsed for 429 responses.
• Batch fetching — fetchMany() crawls a list of URLs concurrently while respecting the same politeness rules, with a permit hook for app-level idempotency.
• Queued entry point — a ready CrawlUrl job that hands fetched documents to a Sink implementation you bind in the container.
• Zero domain coupling — no Eloquent, no models, no schema.

Requirements

• PHP 8.3+
• Laravel 12 or 13

Installation

If you are developing against a local checkout (the recommended workflow for this package), use a path repository in your composer.json:

The service provider is auto-discovered. Publish the configuration when you want to tune the defaults:

Usage

Fetch a single URL

CrawlRequest is a plain DTO:

CrawlResponse carries the raw body and transport metadata:

Fetch many URLs concurrently

fetchMany() crawls a list of requests in waves, applying the same politeness rules, and returns only the attempts that were actually fired:

The permit hook

Pass a permit callable to refuse individual requests right before they are fired — useful for app-level idempotency (e.g. "has this feed already been fetched by another worker?"):

Sink + queued job

The package ships a CrawlUrl queued job and a Sink contract. The job fetches the URL (enforcing politeness) and hands the document to whatever Sink is bound in the container:

Bind it in a service provider:

Then dispatch crawls anywhere:

Note: the job only calls the sink for success / not_modified outcomes. error, throttled and blocked attempts are dropped (log or requeue them yourself if you care).

Configuration

All settings are optional; sensible defaults apply.

How politeness works

• Per-host delay. After a request to a host, a timestamp is written to the cache; further requests to that host are skipped until the delay has passed. A redirect to another host applies the same spacing to the target host.
• Per-host concurrency. A distributed lock per host guarantees at most max_concurrent_per_host in-flight requests, even across multiple queue workers or processes.
• Global concurrency. fetchMany() builds waves of at most max_concurrency requests at a time.
• robots.txt. Fetched once per host (cached for cache_ttl) and only the matching User-agent group (the crawler's own UA or *) is honoured. Unreachable robots.txt never blocks fetching.

Honest headers

The crawler sends a fixed, honest User-Agent, an XML-capable Accept list and Accept-Encoding: gzip. It never impersonates a browser, rotates user agents, or sends Sec-Fetch-* headers.

Testing

The package ships no tests of its own; it is covered by the consuming application's test suite (see the HTTP-fake based tests for rate limiting, robots.txt and conditional-request behaviour).

License

MIT