Download the PHP package oat-sa/extension-tao-task-queue without Composer

Distributed Task Queue

This article describes the functioning of an ASYNC Task Queue.

Install

You can add the Task Queue as a standard TAO extension to your current TAO instance.

Queue component

Queue can work with different types of queue brokers, here two types are to accomplish ASYNC mechanism:

• RdsQueueBroker which stores tasks in RDS.
• SqsQueueBroker which is for using AWS SQS.

Note:

When SqsQueueBroker is used, please make sure that "oat-sa/lib-generis-aws" is included in the main composer.json and you have generis/awsClient.conf.php properly configured.

Weight

A Queue can have a weight. If multiple Queues are in use, this weight will be used for randomly select a Queue to be consumed. For example, if QueueA has weight of 1 and QueueB has weight of 2, then QueueB has about a 66% chance of being selected.

Worker component

Here we have a so called LongRunningWorker which can run unlimited time. It has built-in signal handling for the following actions:

• Shutting down the worker gracefully: SIGTERM/SIGINT/SIGQUIT
• Pausing task processing: SIGUSR2
• Resuming task processing: SIGCONT

Note:

Multiple workers can be run at the same time.

After processing the given task, the worker saves the generated report for the task through the Task Log.

Service setup examples

Multiple Queues settings

In this case we have 3 Queues registered: one of them is using SQS broker, the other two RDS. Every Queue has its own weight (like 90, 30, 10) which will be used at selecting the next queue to be consumed.

And we have two tasks linked to different queues, furthermore the default queue is specified ('background') what will be used for every other tasks not defined in OPTION_TASK_TO_QUEUE_ASSOCIATIONS.

If the queue has not been initialized, meaning the required queue container has not been created yet:

Initializing the queues and the task log container

You can run this script if you want to be sure that the required queues and the task log container are created.

Note:

This script also can be used to change the current queues to use a different queue broker.

• Changing every existing queue to use InMemoryQueueBroker. (Sync Queue)

• Changing every existing queue to use RdsQueueBroker. Option "persistence" is required, "receive" (Maximum amount of tasks that can be received when polling the queue) is optional.

• Changing every existing queue to use SqsQueueBroker. Option "aws-profile" is required, "receive" is optional.

• If you want to apply the settings above for a specific queue, you can add --queue=... option to the command. In the following case, only myQueue will be modified.

• Setting a task selector strategy.

Running a worker

To run a worker, use the following command. It will start a worker for running infinitely and iterating over every registered Queues based in their weights.

If you want the worker running for a dedicated Queue, pass the name of the queue to the command like this:

You can limit the iteration of the worker. It can be used only on a dedicated queue.

If you want to associate specyfic task to new queue you can use this command:

Next time when defined task will be created, it will be assign to specified queue.

Summarize stuck tasks

Execute this command if you want to summarize stuck tasks. Example:

Restart stuck tasks

Execute this command if you want to restart stuck tasks. Example:

Rest API

The task log reports can be viewed/consume using the Application Programming Interface (API). In order to use it please check the swagger file in (doc/taskApi.yml).

Command Line Utility

Besides using the API to check reports of tasks, another way it's using the command line.

This command will show you all the possibilities action the the utility can have.