Download the PHP package stietotalwin/stw-sdk-php-redis-queue without Composer

StieTotalWin Redis Queue

A generic Redis-based queue library for CodeIgniter 4 framework that provides reliable job processing with support for delayed jobs, retries, and failure handling.

Features

• Job Publishing: Publish jobs to Redis queues with optional delays
• Job Consumption: Consume and process jobs from queues
• Retry Mechanism: Automatic retry with exponential backoff for failed jobs
• Multiple Queues: Support for multiple named queues
• Queue Management: Monitor queue statistics, clear queues, and manage jobs
• Bulk Operations: Publish multiple jobs at once
• Failed Job Handling: Track and retry failed jobs
• Job Expiration: Automatic cleanup of expired jobs
• CodeIgniter 4 Integration: Built specifically for CodeIgniter 4 framework
• Unified Configuration: Single Redis configuration for all operations

Requirements

• PHP ^8.5
• CodeIgniter 4 >= 4.7
• Redis server
• Predis client library

Installation

Install via Composer:

Upgrading from 0.1.x to 2.0

2.0 raises the platform floor and fixes three queue-correctness bugs. There are no application API changes — publish(), consume(), markCompleted() and markFailed() keep their existing signatures.

• PHP ^8.5 and CodeIgniter ^4.7 are now required. All implicitly-nullable parameters were made explicit, so the library runs deprecation-free on 8.5. CodeIgniter below 4.7 pulls a laminas/laminas-escaper that refuses to install on PHP 8.5.
• A job that is out of attempts is never run again. Queue recovery previously only recognised a dead job when its stored status read failed. markFailed() sets that status on its in-memory copy only, so an interrupted terminal transition left the stored entry reading processing and the job was requeued on every consume(), forever, appending a dead-letter record each time. Recovery now decides on attempts alone.
• republishStuckJobs() no longer resurrects exhausted jobs. It incremented attempts with no upper bound; it now dead-letters a job that cannot retry.
• A job can only ever be owned by one worker. BZPOPMIN and the validation script are separate round-trips, so recovery could hand an in-flight job to a second worker and the same job would run twice. The script now claims the job atomically and a duplicate claim is dropped.

After upgrading, restart your workers — long-running consumers keep the code they started with, so a composer update alone changes nothing.

Configuration

Create a configuration file app/Config/RedisQueue.php:

Configuration for Cloud Redis (e.g., Upstash)

Usage

Publishing Jobs

Consuming Jobs

Queue Management

Failed Job Management

Job Object Properties

The Job class provides access to job information:

Queue Structure

The library uses the following Redis data structures:

• {queue_name}: Sorted set for pending jobs (score = process_at timestamp)
• {queue_name}:jobs: Hash map storing job data (key = job_id, value = job_json)
• {queue_name}:processing: Sorted set for jobs currently being processed
• {queue_name}:failed: List of failed jobs that exceeded max attempts
• {queue_name}:lost: List of jobs that were lost during processing (for recovery)

Configuration Options

Advanced Features

Job Recovery and Maintenance

The library includes advanced recovery and maintenance features:

• Stuck Job Detection: Automatically detects jobs that have been processing for too long
• Lost Job Recovery: Recovers jobs that were lost due to crashes or interruptions
• Memory-Efficient Cleanup: Uses HSCAN for batch processing to handle large datasets
• Comprehensive Statistics: Provides detailed recovery and cleanup statistics
• Automatic Queue Consistency: Ensures data structure consistency and recovery

Memory Management

The consumer includes built-in memory monitoring:

• Memory Usage Tracking: Monitors PHP memory usage during operations
• Batch Processing: Processes jobs in configurable batches to prevent memory exhaustion
• Explicit Cleanup: Explicitly releases memory after processing large datasets
• Warning System: Logs warnings when memory usage approaches limits

Atomic Operations

The library uses Lua scripts for atomic operations to ensure data consistency:

• Atomic Job Consumption: Jobs are atomically moved from pending to processing
• Consistency Checks: Validates job data exists before processing
• Race Condition Prevention: Prevents multiple consumers from processing the same job

Error Handling

The library provides robust error handling:

• Automatic Retries: Failed jobs are automatically retried with exponential backoff
• Failed Job Storage: Jobs that exceed max attempts are stored in a failed jobs list
• Error Messages: Exception messages are captured and stored with failed jobs
• Job Expiration: Old jobs are automatically cleaned up to prevent memory leaks

Best Practices

1. Use Specific Queue Names: Create separate queues for different job types
2. Handle Exceptions: Always wrap job processing in try-catch blocks
3. Monitor Failed Jobs: Regularly check and handle failed jobs
4. Set Appropriate TTL: Configure job TTL based on your use case
5. Clean Up Regularly: Use the cleanup functionality to remove expired jobs
6. Use Delays Wisely: For time-sensitive jobs, use delays instead of immediate processing
7. Secure Connections: Use TLS scheme for production Redis connections
8. Connection Pooling: Consider using connection pooling for high-throughput applications
9. Monitor Memory Usage: Enable detailed logging during development to monitor memory usage
10. Run Recovery Operations: Periodically check and run recovery operations for stuck jobs
11. Batch Processing: Use appropriate batch sizes for cleanup operations on large queues
12. Data Validation: Always validate and decode JSON data from jobs before processing

License

This library is licensed under the MIT License. See the LICENSE file for details.

Support

For issues and questions, please contact: [email protected]