Download the PHP package changhorizon/crawler without Composer

On this page you can find all versions of the php package changhorizon/crawler. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.

FAQ

After the download, you have to make one include require_once('vendor/autoload.php');. After that you have to import the classes with use statements.

Example:
If you use only one package a project is not needed. But if you use more then one package, without a project it is not possible to import the classes with use statements.

In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
Some PHP packages are not free to download and because of that hosted in private repositories. In this case some credentials are needed to access such packages. Please use the auth.json textarea to insert credentials, if a package is coming from a private repository. You can look here for more information.

  • Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
  • To use Composer is sometimes complicated. Especially for beginners.
  • Composer needs much resources. Sometimes they are not available on a simple webspace.
  • If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
  • Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Please rate this library. Is it a good library?

Example code of changhorizon/crawler

Informations about the package crawler

Crawler

Effortless recursive web crawling for comprehensive site indexing

License Latest Version PHP Version Static Analysis Tests codecov CI

A high-performance PHP web crawler library designed to recursively scrape all pages of a single website.This library is ideal for building site search, content analysis, SEO data collection, and similar projects. It is designed to be simple, extensible, and serve as a solid foundation for web crawling and data extraction.

✨ 特性

📦 安装

📂 目录结构

🚀 用法示例

示例 1：基础爬取示例

示例 2：自定义爬取间隔

📐 接口说明

Crawler::__construct

构造函数，创建一个爬虫实例。

Crawler::run

启动爬虫任务。

Crawler::fetchUrlContent

获取 HTML 页面正文内容，过滤掉响应头，若请求无效返回 null。

Crawler::scrapePage

抓取并解析页面，存储内容，并递归抓取内部链接。

Crawler::extractLinks

从 HTML 中提取合法的绝对链接。

Crawler::resolveUrl

将相对链接解析为绝对 URL，仅保留与起始 URL 同源的链接。

ChangHorizon\Crawler\Storage\DocumentStorageInterface

定义用于文档存储的接口，包含存储与状态检测方法。

ChangHorizon\Crawler\Http\HttpClientInterface

定义用于发起 HTTP 请求的接口。

备注： 所有接口实现均来自包内组件，适合用于扩展和替换。爬虫本体逻辑通过 DocumentStorageInterface 和 HttpClientInterface 解耦，便于测试与复用。

🔍 静态分析

使用 PHPStan 工具进行静态分析，确保代码的质量和一致性：

🎯 代码风格

使用 PHP-CS-Fixer 工具检查代码风格：

使用 PHP-CS-Fixer 工具自动修复代码风格问题：

✅ 单元测试

执行 PHPUnit 单元测试：

执行 PHPUnit 单元测试并生成代码覆盖率报告：

🤝 贡献指南

欢迎 Issue 与 PR，建议遵循以下流程：

  1. Fork 仓库
  2. 创建新分支进行开发
  3. 提交 PR 前请确保测试通过、风格一致
  4. 提交详细描述

📜 License

MIT License. See the LICENSE file for details.

All versions of crawler with dependencies

PHP Build Version
Package Version
Requires php Version >=8.2
Composer command for our command line client (download client) This client runs in each environment. You don't need a specific PHP version etc. The first 20 API calls are free. Standard composer command

The package changhorizon/crawler contains the following files

Loading the files please wait ...