Download the PHP package kynx/swoole-processor without Composer

kynx/swoole-processor

Run CLI batch jobs in coroutines across multiple processes.

Based on Swoole, the processor is ideal for running a large number of IO-intensive operations.

Install

Use

Create a JobProvider class that returns Job objects containing the payload you want to process:

Create a Worker class to handle the jobs:

If required, create a CompletionHandler to do any post-processing:

Create a script to run the jobs in parallel:

If you don't need to handle completion, omit the fourth parameter.

How it works

The JobProvider runs in its own process, and the resulting jobs are fired at a Swoole server listening on a local socket. Both it and the the CompletionHandler are blocking. If the JobProvider returns a large number of jobs or performs time-consuming operations, return a Generator so jobs can be started as soon as possible. The CompletionHandler should be fast.

The server spawns a number of processes to handle the jobs - by default one less than the number of CPU cores detected. The process runs your Worker inside a coroutine, ensuring IO operations do not block. Because of this, ensure all IO uses a Connection Pool, covered in more detail below.

You can control the number of simultaneous jobs the processor will spawn by setting the concurrency parameter on the the Config you pass to the constructor. It defaults to 10, with a maximum of workers x maxCoroutines.

Caveats

• Job payloads and results must be serializable and, but default, together should be less than 2M when serialized
• A Worker should be stateless. If you need share data between workers, use a Table.
• Uncaught exceptions will crash the process, causing it it re-spawn. For this reason your Worker::run() is called inside a try ... catch block. However the exception is discarded: if you care about where your program is going wrong, catch all exceptions inside your Worker and handle them yourself.