Download the PHP package blogdaren/phpcreeper without Composer
On this page you can find all versions of the php package blogdaren/phpcreeper. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Download blogdaren/phpcreeper
More information about blogdaren/phpcreeper
Files in blogdaren/phpcreeper
Package phpcreeper
Short Description A new generation of multi-process async event-driven spider engine based on Workerman
License MIT
Homepage http://www.phpcreeper.com
Informations about the package phpcreeper
PHPCreeper
What is it
PHPCreeper is a new generation of multi-process asynchronous event-driven spider engine based on workerman.
- Focus on efficient agile development, and make the crawler job become more easy
- Solve the performance and scalability bottlenecks of traditional crawler frameworks
- Take advantage of crawling fully in Multi-Process + Distributed + Separated environment
- Support headless browser which can execute JavaScript codes for crawling dynamic pages
爬山虎是基于workerman开发的全新一代多进程异步事件驱动型PHP爬虫引擎, 它有助于:
- 专注于高效敏捷开发,让爬取工作变得更加简单。
- 解决传统型PHP爬虫框架的性能和扩展瓶颈问题。
- 充分发挥多进程+分布式+分离式部署环境下的爬取优势。
- 支持无头浏览器即支持运行JavaScript代码及其渲染页。
Documentation
The chinese document is relatively complete, and the english document will be kept up-to-date constantly here.
注意: 爬山虎中文开发文档相对比较完善,各位小伙伴直接点击下方链接阅读即可.
- 爬山虎中文官方网站:http://www.phpcreeper.com
- 中文开发文档主节点:http://www.phpcreeper.com/docs/
- 中文开发文档备节点:http://www.blogdaren.com/docs/
- 爬山虎是一个免费开源的佛系爬虫项目,欢迎小星星Star支持,让更多的人发现、使用并受益。
- 爬山虎源码根目录下有一个
Examples/start.php
样例脚本,开发之前建议先阅读它而后运行它。 - 爬山虎提供的例子如果未能按照预期工作,请检查修改爬取规则,因为源站DOM极可能更新了。
技术交流
- 下方绿色二维码为微信交流群:phpcreeper 【进群之前需先加此专属微信并备明来意或附上备注:爬山虎】
- 若是奔着虎哥的原创视频《深入PHP内核源码》而来,务必添加专属微信方可获得配套教程文档,弥足珍贵。
- 微信群主要围绕 爬山虎 和 workerman 和 深入PHP内核源码 开展技术交流,观看PHP内核视频请移步至B站。
Screenshot
Features
- Inherit almost all features from workerman
- Support headless browser for crawling dynamic pages
- Support router to different parser by the task type
- Support Crontab-Jobs similar to Linux-Crontab
- Support distributed and separated deployment
- Support agile development with PHPCreeper-Application
- Use PHPQuery as the elegant content extractor
- With high performance and strong scalability
Prerequisites
- PHP_VERSION ≥ 7.0.0 (Better to choose PHP 7.4+ for some compatibility reasons)
- A POSIX compatible OS (Linux、OSX、BSD)
- POSIX extension for PHP (Required)
- PCNTL extension for PHP (Required)
- REDIS extension for PHP (Optional, note that
predis
will be the default redis client since v1.4.2) - EVENT extension for PHP (Optional, it's strongly recommended to install for better performance)
- 简单的说:只要能跑起来workerman那就能跑起来PHPCreeper,所以安装要求和workerman完全一致。
- POSIX扩展和PCNTL扩展是必选项:PHP发行包一般都会默认安装这两个扩展,若没有请自行编译安装。
- EVENT扩展是可选项:建议最好安装,这是提升各路性能的一个主要支撑;另注意需要优化Linux内核。
- REDIS扩展是可选项: 注意:v1.4.2版本之后,引擎默认采用predis客户端,所以不再强依赖REDIS扩展。
Installation
The recommended way to install PHPCreeper is through Composer.
Usage: NOT Depend On The PHPCreeper Application Framework
Firstly, there is another matched Application Framework named PHPCreeper-Application which is published simultaneously for your development convenience, although this framework is not necessary, we strongly recommend that you use it which will greatly improve your job efficiency. Besides, we can also write the code which NOT depends on the framework, it is also easy to make it.
Next let's take an example to show how to capture the weather in Washington in 7 days
:(See Full Demo Here)
Now, save the example code above to a file and name it to be weather.php
as a startup script, then run it like this:
Usage: Depend On The PHPCreeper Application Framework
If u wanna develop the app based on the PHPCreeper Application Framework
or see more configuration
,click here
How to set extractor rule
- Per URL config item match a unique rule config item, and the rule_name must be one-to-one correspondence
- The type of rule value must be Array
- For a single task, the depth of the corresponding rule item, that is, the depth of the array, can only be 2
-
For multi task, the depth of the corresponding rule item, that is, the depth of the array, can only be 3
-
rule_name
you should give an unique rule name for each task, so that we can easily index the data that we want, if you leave it empty, it will usemd5($task_id)
notmd5($task_url)
which has potential pitfalls as the unique rule name since v1.6.0 -
selector
selector must be provided, or it will be ignored, just like jQuery selector, its value can be like#idName
or.className
orHtml Element
and so on. -
action
default value istext
, indicates what action we should take, the value can be one of the following:
text
: used to get the inner text of html element
html
: used to get the inner text with tags of html element
attr
: used to get the attrbute value of html element
【Attention: the real value shoud be the attribute likesrc
、href
etc, NOTattr
itself】
css
: used specially to get the style attribute of html element, and return as an array form
【*Attention: support also more variant format like `css:、
css:prop1,prop2,...propN`**】 -
range
used to narrow down the entries to only those that match, just like jQuery selector, the value can be like#idName
or.className
orHtml Element
and so on. -
callback
you can trigger acallback string
orcallback function
here, but remember to return the data expected.callback string:
it is recommended to use and semantically equivalent to the PHP native callback function.
callback function:
note that you should usecallback string
instead of, because PHP native callback function may work unexpectedly in communication across multi-process environments.callback function
Use Database
PHPCreeper wrappers a lightweight database like Medoo style, please visit the Medoo official site if you wanna know more about its usage. now we just need to find out how to get the DBO, as a matter of fact, it is very simple:
First configure the database.php
then add the code listed below:
Now we can get DBO and start the query or the other operation as you like:
Available commands
Note that all the commands in PHPCreeper
can only run on the command line,
and you must write a global entry startup script whose name
assumed to be start.php
before you start any crawling jobs, but if you use the
PHPCreeper-Application
framework for your development, it will automatically
help you generate all the startup scripts including global we need.
Related links and thanks
LICENSE
PHPCreeper is released under the MIT License.
DISCLAIMER
Please DON'T use PHPCreeper for any businesses which are NOT PERMITTED BY LAW in your country.
Please comply with the spider protocol for friendly use of PHPCreeper, if you choose to use PHPCreeper,
you will comply with this agreement. I take no warranty or responsibility for this code. Use at your own risk.
All versions of phpcreeper with dependencies
workerman/workerman Version >=3.5.0,<5.0.0
blogdaren/configurator Version ^1.0
blogdaren/logger Version ^1.1.2
guzzlehttp/guzzle Version ^6.4 || ^7.0
predis/predis Version 2.0.*
chrome-php/chrome Version ^1.11