Download the PHP package idanoo/php-resque without Composer

php-resque: PHP Background (Resque) Worker

Resque is a Redis-backed library for creating background jobs, placing those jobs on one or more queues, and processing them later.

Background

Resque was developed by the folks at GitHub and written in Ruby. What you're seeing here is an almost direct port of the Resque worker and enqueue system to PHP.

For more information on Resque, visit the official GitHub project: https://github.com/resque/resque

For further information, see the launch post on the GitHub blog: http://github.com/blog/542-introducing-resque

The PHP port does NOT include its own web interface for viewing queue stats, as the data is stored in the exact same expected format as the Ruby version of Resque.

The PHP port provides much the same features as the Ruby version:

• Workers can be distributed between multiple machines
• Includes support for priorities (queues)
• Resilient to memory leaks (forking)
• Expects failure

It also supports the following additional features:

• Has the ability to track the status of jobs
• Will mark a job as failed, if a forked child running a job does not exit with a status code as 0
• Has built in support for setUp and tearDown methods, called pre and post jobs

On top of the original fork (chrisboulton/php-resque) I have added:

• Custom log levels
• PHP7.0+ compatibility

Requirements

• PHP 8.4+
• phpredis
• Redis 2.2+

Getting Started

The easiest way to work with php-resque is when it's installed as a Composer package inside your project.

If you're not familiar with Composer, please see http://getcomposer.org/.

1. Run composer require idanoo/php-resque.

2. If you haven't already, add the Composer autoload to your project's initialization file. (example)

Jobs

Queueing Jobs

Jobs are queued as follows:

Defining Jobs

Each job should be in its own class, and include a perform method.

When the job is run, the class will be instantiated and any arguments will be set as an array on the instantiated object, and are accessible via $this->args.

Any exception thrown by a job will result in the job failing - be careful here and make sure you handle the exceptions that shouldn't result in a job failing.

Jobs can also have setUp and tearDown methods. If a setUp method is defined, it will be called before the perform method is run. The tearDown method, if defined, will be called after the job finishes.

Dequeueing Jobs

This method can be used to conveniently remove a job from a queue.

If no jobs are given, this method will dequeue all jobs matching the provided queue.

Workers

Workers work in the exact same way as the Ruby workers. For complete documentation on workers, see the original documentation.

A basic "up-and-running" bin/resque file is included that sets up a running worker environment. (vendor/bin/resque when installed via Composer)

The exception to the similarities with the Ruby version of resque is how a worker is initially setup. To work under all environments, not having a single environment such as with Ruby, the PHP port makes no assumptions about your setup.

To start a worker, it's very similar to the Ruby version:

It's your responsibility to tell the worker which file to include to get your application underway. You do so by setting the APP_INCLUDE environment variable:

Pro tip: Using Composer? More than likely, you don't need to worry about APP_INCLUDE, because hopefully Composer is responsible for autoloading your application too!

Getting your application underway also includes telling the worker your job classes, by means of either an autoloader or including them.

Alternately, you can always include('bin/resque') from your application and skip setting APP_INCLUDE altogether. Just be sure the various environment variables are set (setenv) before you do.

Logging

The port supports the same environment variables for logging to STDOUT. Setting LOGLEVEL will print different logs depending on levels. Valid loglevels are listed below with an example. Default LOGLEVEL is WARNING.

Priorities and Queue Lists

Similarly, priority and queue list functionality works exactly the same as the Ruby workers. Multiple queues should be separated with a comma, and the order that they're supplied in is the order that they're checked in.

As per the original example:

The file_serve queue will always be checked for new jobs on each iteration before the warm_cache queue is checked.

Running All Queues

All queues are supported in the same manner and processed in alphabetical order:

Running Multiple Workers

Multiple workers can be launched simultaneously by supplying the COUNT environment variable:

Be aware, however, that each worker is its own fork, and the original process will shut down as soon as it has spawned COUNT forks. If you need to keep track of your workers using an external application such as monit, you'll need to work around this limitation.

Custom prefix

When you have multiple apps using the same Redis database it is better to use a custom prefix to separate the Resque data:

Redis Backend & Authentication

The Redis connection is configured via the REDIS_BACKEND environment variable. It accepts either a simple host:port value or a DSN-style URI, which is the recommended format for production as it lets you supply authentication and connection options:

Supported DSN format:

Notes:

• Supply just a password (redis://:password@host) to authenticate as the default user with AUTH password — this is what requirepass expects.
• Supply both parts (redis://user:password@host) to authenticate a Redis 6+ ACL user with AUTH user password. A username on its own is ignored.
• The username and password are percent-decoded, so credentials containing reserved characters must be encoded: p@ss:word becomes p%40ss%3Aword.
• Credentials are held by the driver and replayed automatically if the connection drops and is re-established.
• Always run Redis with authentication enabled and restrict network access in production. php-resque connects without a password if none is supplied, so an unprotected Redis is reachable by anything that can route to it.
• Supported schemes are redis, tcp, rediss, tls, ssl, and unix:// (for a socket path).

Dropped connections

A command that fails because the connection died — a managed-Redis failover, an idle connection reaped by the server's timeout, a node restart — is retried once on a fresh connection, with credentials and the selected database replayed. Without this a single failover of, say, an ElastiCache or Valkey primary takes every worker down with Error communicating with Redis: read error on connection.

The retry means a write whose reply was lost may be applied twice, which is the same at-least-once guarantee Resque already gives a job that exits dirty. Commands where a replay would be wrong are never retried: anything inside a MULTI/pipeline, WATCH/UNWATCH, and the subscribe commands. Use the max_connect_retries DSN option to also retry the initial dial.

Persistent connections

The persistent DSN option keeps the socket open instead of reconnecting, but it interacts badly with per-job forking: Credis skips an unforced close() while persistent is set, so before 3.5.0 the socket survived the pre-fork disconnect and parent and child both wrote to it. Interleaved commands on one socket desynchronise the protocol and surface as read error on connection — usually on the parent's next BLPOP. php-resque now forces the close, but FORK_PER_JOB=0 avoids the fork entirely and is the better pairing for persistent.

TLS

Use the rediss:// scheme (or tls:// / ssl://) to connect over an encrypted channel — required by managed services such as AWS ElastiCache with in-transit encryption enabled:

The certificate chain is verified against the system CA bundle by default. TLS behaviour is tuned with tls_-prefixed DSN options, each of which maps to the PHP SSL context option of the same name with the prefix removed:

Boolean options accept 0, false, off, no or an empty value as false and anything else as true. An unrecognised tls_ option is rejected rather than ignored, so a typo cannot silently leave verification disabled.

Note on ACL users: when the redis PHP extension is not installed (or is older than 5.3.0) php-resque falls back to Credis' pure-PHP client. On one of that client's reconnect paths — a command issued after the server has closed an idle connection — it re-sends AUTH with the password only, dropping the username. Against an ACL user that authenticates as the default user instead, or fails outright. If you use user:password credentials, install the redis extension (5.3.0 or newer) so the extension's own connection handling is used.

Forking

Similarly to the Ruby versions, supported platforms will immediately fork after picking up a job. The forked child will exit as soon as the job finishes.

The difference with php-resque is that if a forked child does not exit nicely (PHP error or such), php-resque will automatically fail the job.

Set FORK_PER_JOB=0 to run jobs inline in the worker instead:

$ QUEUE=file_serve FORK_PER_JOB=0 bin/resque

0, false, no, off and an empty value all disable forking; anything else (or leaving the variable unset) keeps the default fork-per-job behaviour.

Forking costs two Redis connections per job — Resque::fork() drops the worker's connection before forking, then the child and the parent each open a new one. When connection setup is expensive (TLS handshake plus AUTH, as on an encrypted ElastiCache/Valkey endpoint) and job throughput is high, that connection churn can dominate server CPU. Running inline keeps one connection alive for the life of the worker.

What you give up:

• A job that exits dirty (fatal error, OOM, exit()) takes the worker down with it instead of being failed as a DirtyExitException. Run workers under a supervisor that restarts them.
• USR1 and TERM can no longer interrupt a job that is already running. QUIT still works, and TERM still stops the worker once the current job finishes.
• Jobs share process state — memory is not reclaimed between jobs, and anything a job mutates in a static or global is visible to the next one.

beforeFork and afterFork still fire in both modes, so listeners that re-establish per-job resources keep working.

Signals

Signals also work on supported platforms exactly as in the Ruby version of Resque:

• QUIT - Wait for job to finish processing then exit
• TERM / INT - Immediately kill job then exit
• USR1 - Immediately kill job but don't exit
• USR2 - Pause worker, no new jobs will be processed
• CONT - Resume worker.

Process Titles/Statuses

The Ruby version of Resque has a nifty feature whereby the process title of the worker is updated to indicate what the worker is doing, and any forked children also set their process title with the job being run. This helps identify running processes on the server and their resque status.

PHP does not have this functionality by default until 5.5.

A PECL module (http://pecl.php.net/package/proctitle) exists that adds this functionality to PHP before 5.5, so if you'd like process titles updated, install the PECL module as well. php-resque will automatically detect and use it.

Event/Hook System

php-resque has a basic event system that can be used by your application to customize how some of the php-resque internals behave.

You listen in on events (as listed below) by registering with \Resque\Event and supplying a callback that you would like triggered when the event is raised:

[callback] may be anything in PHP that is callable by call_user_func_array:

• A string with the name of a function
• An array containing an object and method to call
• An array containing an object and a static method to call
• A closure

Events may pass arguments (documented below), so your callback should accept these arguments.

You can stop listening to an event by calling \Resque\Event::stopListening with the same arguments supplied to \Resque\Event::listen.

It is up to your application to register event listeners. When enqueuing events in your application, it should be as easy as making sure php-resque is loaded and calling \Resque\Event::listen.

When running workers, if you run workers via the default bin/resque script, your APP_INCLUDE script should initialize and register any listeners required for operation. If you have rolled your own worker manager, then it is again your responsibility to register listeners.

Events

beforeFirstFork

Called once, as a worker initializes. Argument passed is the instance of \Resque\Worker that was just initialized.

beforeFork

Called before php-resque forks to run a job. Argument passed contains the instance of \Resque\Job for the job about to be run.

beforeFork is triggered in the parent process. Any changes made will be permanent for as long as the worker lives.

afterFork

Called after php-resque forks to run a job (but before the job is run). Argument passed contains the instance of \Resque\Job for the job about to be run.

afterFork is triggered in the child process after forking out to complete a job. Any changes made will only live as long as the job is being processed.

beforePerform

Called before the setUp and perform methods on a job are run. Argument passed contains the instance of \Resque\Job for the job about to be run.

You can prevent execution of the job by throwing an exception of \Resque\Job_DontPerform. Any other exceptions thrown will be treated as if they were thrown in a job, causing the job to fail.

afterPerform

Called after the perform and tearDown methods on a job are run. Argument passed contains the instance of \Resque\Job that was just run.

Any exceptions thrown will be treated as if they were thrown in a job, causing the job to be marked as having failed.

onFailure

Called whenever a job fails. Arguments passed (in this order) include:

• Exception - The exception that was thrown when the job failed
• \Resque\Job - The job that failed

beforeEnqueue

Called immediately before a job is enqueued using the Resque::enqueue method. Arguments passed (in this order) include:

• Class - string containing the name of the job to be enqueued
• Arguments - array of arguments for the job
• Queue - string containing the name of the queue the job is to be enqueued in
• ID - string containing the token of the job to be enqueued

You can prevent enqueing of the job by throwing an exception of \Resque\Job_DontCreate.

afterEnqueue

Called after a job has been queued using the Resque::enqueue method. Arguments passed (in this order) include:

• Class - string containing the name of scheduled job
• Arguments - array of arguments supplied to the job
• Queue - string containing the name of the queue the job was added to
• ID - string containing the new token of the enqueued job

Supervisor Configuration

You may like to run php-resque on a supervisor task to manage the processes. The following is a default config that can be modified to suit.

Issues:

• Restarting worker doesn't always make it use updated code, I find on a dev-environment issuing the following command works well to restart everything. sudo /etc/init.d/supervisor force-stop && sleep 1 && sudo /etc/init.d/supervisor restart

Step-By-Step

For a more in-depth look at what php-resque does under the hood (without needing to directly examine the code), have a look at HOWITWORKS.md.

Contributors

Current Maintainers

• @idanoo

Past Maintainer / Forked From

• @chrisboulton

Others

• @acinader
• @ajbonner
• @andrewjshults
• @atorres757
• @benjisg
• @cballou
• @chaitanyakuber
• @charly22
• @CyrilMazur
• @d11wtq
• @danhunsaker
• @dceballos
• @ebernhardson
• @hlegius
• @hobodave
• @humancopy
• @iskandar
• @JesseObrien
• @jjfrey
• @jmathai
• @joshhawthorne
• @KevBurnsJr
• @lboynton
• @maetl
• @matteosister
• @MattHeath
• @mickhrmweb
• @Olden
• @patrickbajao
• @pedroarnal
• @ptrofimov
• @rajibahmed
• @richardkmiller
• @Rockstar04
• @ruudk
• @salimane
• @scragg0x
• @scraton
• @thedotedge
• @tonypiper
• @trimbletodd
• @warezthebeef