Download the PHP package mmucklo/queue-bundle without Composer

DtcQueueBundle

Allow symfony developers to create background job as easily as: $worker->later()->process(1,2,3)

6.0 Release

See changes

Upgrading from 5.0: see UPGRADING-6.0.md

Supported Queues

• MongoDB via Doctrine-ODM
• Mysql / Doctrine 2 supported databases via Doctrine-ORM
• Beanstalkd via pheanstalk
• RabbitMQ via php-amqplib
• Redis support via Predis or PhpRedis, or through SncRedisBundle

Introduction

This bundle provides a way to easily create and manage queued background jobs

Basic Features:

• Ease of Use
  • Kickoff background tasks with a line of code or two
  • Easily add background worker services
    • Turn any code into background task with a few lines
• Atomic operation for jobs
  • For ORM-based queues this is done without relying on transactions.
• Admin interface
  • Web-based Admin interface with an optional performance graph
• Command Line Interface
  • Commands to run, manage and debug jobs from console
• Job Archival
  • ORM and ODM managers have built-in job-archival for finished jobs
• Logs errors from worker
• Various safety checks for things such as stalled jobs, exception jobs
  • Allows for reseting stalled and exception jobs via console commands
• Built in Event Dispatcher

Job-specific Features:

• Auto-retry on failure, exception
  • If a job exits with a failure code, it can auto-retry
  • Same for Exception if desired
• Priority
  • Jobs can have levels of priority so that higher priority jobs can get processed first even if they were added
  • to the queue later.
• Future Jobs (ODM / ORM / Redis)
  • Jobs can be scheduled to run at some time in the future
• Batch
  • Jobs can be "batched" so that only one job runs, even if multiple are queued of the same type
• Expires
  • Jobs can have an "expires" time so that they wont run after a certain point
    • (useful if the queue gets backed up and a job is worthless after a certain time)
• Stalls (ODM / ORM)
  • Jobs that crash the interpreter, or get terminated for some other reason can be detected
    • These can be re-queued to run in the future.

Installation

Symfony 2/3

see /Resources/doc/symfony2-3.md

Symfony 4/5

see /Resources/doc/symfony4-5.md

Troubleshooting

see /Resources/doc/troubleshooting.md

Usage

Create a worker class that will work on the background job.

Example:

• src/Worker/Fibonacci.php: (symfony 4/5)
• src/AppBundle/Worker/Fibonacci.php: (symfony 2/3)

Create a DI service for the job, and tag it as a background worker.

YAML:

Symfony 5, 4 and 3.3, 3.4:

Symfony 2, and 3.0, 3.1, 3.2:

XML:

Create a job

Simple examples:

Command line:

Code:

Create Jobs - Additional Information

For further instructions on creating jobs, including how to create a job from the command line using json-encoded arguments, see:

/Resources/doc/create-job.md

Running Jobs

It's recommended that you background the following console commands

Pruning Jobs

For ODM and ORM based stores, the archive tables and the regular job table (queue) can require periodic pruning.

The regular job table is for waiting and running jobs. If a job throws an exception that can't be caught or the process segfaults, machine crashes, etc. then jobs which never finished can remain in the job queue in the "Running" state.

The archive table is where finished and errored jobs end up after execution; this table can grow indefinitely.

For Mongo in production, it may be prudent to use a capped collection or TTL Indexes

For Mysql you could create an event to delete data periodically.

Nevertheless there are also several commands that exist that do similarly (and could be put into a periodic cron job as well):

Debugging

These commands may help with debugging issues with the queue:

Tracking Runs

Each job run can be tracked in a table in an ORM / ODM backed datastore.

Ways to configure: app/config/config.yml: (symfony 2/3) __config/packages/dtc_queue.yaml:__ (symfony 4/5)

MongoDB DocumentManager

Change the document manager

app/config/config.yml: (symfony 2/3) __config/packages/dtc_queue.yaml:__ (symfony 4/5)

Mysql / ORM Setup

As of 4.0, ORM requires the bcmath extension to be enabled

app/config/config.yml: (symfony 2/3) __config/packages/dtc_queue.yaml:__ (symfony 4/5)

Change the EntityManager:

NOTE: You may need to add DtcQueueBundle to your mappings section in config.yml if auto_mapping is not enabled

Note on NON-ORM Setups:

If you plan on using ODM or Redis or another configuration, but you have Doctrine ORM enabled elsewhere, it's recommended that you use the schema_filter configuration parameter so that schema dumps and/or migration diffs don't pickup those tables (see issue #77).

E.g.

_(if you already have a schemafilter, you can just add the "dtc_" prefix to it.)

Beanstalk Configuration

app/config/config.yml: (symfony 2/3) __config/packages/dtc_queue.yaml:__ (symfony 4/5)

RabbitMQ Configuration

app/config/config.yml: (symfony 2/3) __config/packages/dtc_queue.yaml:__ (symfony 4/5)

Redis Configuration

app/config/config.yml: (symfony 2/3) __config/packages/dtc_queue.yaml:__ (symfony 4/5)

Custom Jobs and Managers

app/config/config.yml: (symfony 2/3) __config/packages/dtc_queue.yaml:__ (symfony 4/5)

Rename the Database or Table Name

1) Extend the following:

or

(Depending on whether you're using MongoDB or an ORM)

2) Change the parameters on the class appropriately

3) Add the new class(es) to config.yml

Job Event Subscriber

It's useful to listen to event in a long running script to clear doctrine manager or send email about status of a job. To add a job event subscriber, create a new service with tag: dtc_queue.event_subscriber:

ClearManagerSubscriber.php

Running as upstart service:

1. Create the following file in /etc/init/. PHP is terrible at memory management and garbage collection: to deal with out of memory issues, run 20 jobs at a time. (Or a manageable job size)

2. Reload config: sudo initctl reload-configuration
3. Start the script: sudo start queue

Admin

NOTE: ORM And ODM (MongoDB) require mmucklo/grid-bundle in order to view the jobs/runs admin page.

You can register admin routes to see queue status. In your routing.yml file, add the following:

Testing

You can run unittest by typing bin/phpunit in source folder. If you want to run integration testing with Mongodb, you need to set up Mongodb server on localhost and run:

If you want to run Beanstalkd integration testing, you need to run a local, empty instance of beanstalkd for testing.

Full Configuration

See /Resources/doc/full-configuration.md

License

This bundle is under the MIT license.

Credit

Originally written by @dtee Enhanced and maintained by @mmucklo