PHP download

Download the PHP package tonsoo/php-crawler without Composer

On this page you can find all versions of the php package tonsoo/php-crawler. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.

FAQ

After the download, you have to make one include require_once('vendor/autoload.php');. After that you have to import the classes with use statements.

Example:

If you use only one package a project is not needed. But if you use more then one package, without a project it is not possible to import the classes with use statements.

In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.

Some PHP packages are not free to download and because of that hosted in private repositories. In this case some credentials are needed to access such packages. Please use the auth.json textarea to insert credentials, if a package is coming from a private repository. You can look here for more information.

Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
To use Composer is sometimes complicated. Especially for beginners.
Composer needs much resources. Sometimes they are not available on a simple webspace.
If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.

Please rate this library. Is it a good library?

Example code of tonsoo/php-crawler

Informations about the package php-crawler

Sitemap Generator Crawler

A small, dependency-light PHP crawler that walks a site and generates XML sitemaps. It follows links, respects meta robots directives, and ships with a sitemap extension that can write a single sitemap or rotate into multiple files with an index.

Requirements

PHP 8.4+
Extensions: ext-dom, ext-curl, ext-xmlwriter

Installation

Quick Start

This will crawl https://example.com, write sitemap.xml (or sitemap-2.xml, sitemap-3.xml, etc.), and produce a sitemap-index.xml once multiple sitemap files are created.

Crawler Configuration

The crawler is configured via a fluent API on Crawler:

What these options do

displayCrawls(true): toggles crawl logging (currently not used by the built-in logger).
displayMemoryInfo(true): toggles memory logging (currently not used by the built-in logger).
respectNoIndex(true): honors <meta name="robots" content="noindex"> (default: true).
respectNoFollow(true): honors <meta name="robots" content="nofollow"> (default: true).
respectCanonical(true): uses the canonical URL for link resolution (default: true).
preserveScheme(true): stays on the same scheme (http vs https) (default: true).
preserveHost(true): stays on the same host (default: true).
maxPages(5000): stops after a page limit (default: null = unlimited).

Sitemap Generation

Single sitemap

Rotating sitemap + index

Notes:

RotatingSitemapWriter requires the directory to already exist.
The index file is written only when more than one sitemap file is created.
The index stores the sitemap filenames (relative paths), not absolute URLs.

Events

You can subscribe to crawler events to observe or extend behavior:

Custom HTTP Client, Logger, and Analyzer

You can plug in your own implementations:

Defaults:

HTTP client: CurlHttpClient (follows redirects, 4s connect/total timeout, custom UA string).
Logger: ConsoleLogger (timestamps to stdout).
Analyzer: DomDocumentPageAnalyzer (DOM + XPath).

Interfaces to implement:

HttpClientInterface::fetch(string $url): Result
LoggerInterface::log(string $message): void
PageAnalyzerInterface::analyze(Result $result, bool $respectNoIndex, bool $respectNoFollow): PageAnalysis

Error Handling

If maxPages is set and the crawler reaches the limit, it throws LimitExceededException after finishing the crawl loop:

Crawling Behavior

The crawler only processes pages that return an HTML body with a text/html content type. If a page has no HTML body or a non-HTML content type, it is skipped and the corresponding event is emitted.

The crawler collects links from <a href="..."> elements and normalizes them. It will:

Resolve relative URLs against the current page
Drop fragments (the #... part)
Ignore non-HTTP(S) schemes
Optionally restrict links by host and scheme
Optionally respect noindex / nofollow meta tags (from <meta name="robots">)
Use canonical URLs when enabled

This crawler does not parse robots.txt.

Example Script

See examples/crawler.php for a full working example.

All versions of php-crawler with dependencies

PHP Build Version

Package Version

Version v1.0.1 Release 23. Feb 2026
create-project require 0 people chose require and
0 people chose create-project.

Download

Download latest version of php-crawler from vendor tonsoo

Requires php Version ^8.4
ext-dom Version *
nesbot/carbon Version ^3.11
league/uri Version ^7.8
ext-curl Version *
ext-xmlwriter Version *

Composer command for our command line client (download client) This client runs in each environment. You don't need a specific PHP version etc. The first 20 API calls are free. Standard composer command

The package tonsoo/php-crawler contains the following files

Loading the files please wait ...