Download the PHP package yangusik/laravel-balanced-queue without Composer

Laravel Balanced Queue

A Laravel package for queue management with load balancing between partitions (user groups). Perfect for scenarios where you need fair job distribution and concurrency control per user/tenant.

Problem Solved

Imagine you have an AI generation service where users can submit unlimited tasks. Without balanced queuing:

• One user can flood the queue and block everyone else
• No control over how many concurrent tasks a single user can run
• Resource-heavy users can exhaust API rate limits

Laravel Balanced Queue solves this by:

• Distributing jobs fairly across all users (round-robin)
• Limiting concurrent jobs per user (e.g., max 2 AI generations per user)
• Never rejecting jobs - they queue up and execute eventually
• Preventing single users from monopolizing workers

How It Works

The Problem

The Solution

Balanced Queue partitions jobs by user and rotates between them:

Strategy Comparison

Round-Robin Strategy (recommended) — strict rotation:

Random Strategy — unpredictable but fast:

Smart Strategy — prioritizes smaller queues:

Concurrency Limiting

With max_concurrent: 2 per partition:

Installation

Publish the configuration:

Quick Start

Step 1: Add Queue Connection

Add to config/queue.php:

Step 2: Create a Job

Step 3: Dispatch Jobs

Step 4: Run Workers

That's it! Jobs are now distributed fairly with max 2 concurrent per user.

Laravel Horizon Integration

Configuration

Add a supervisor for balanced queue in config/horizon.php:

Important: What Works and What Doesn't with Horizon

Why? Balanced Queue uses a different Redis key structure (partitioned queues) than standard Laravel queues. Horizon expects jobs in queues:{name} but we store them in balanced-queue:queues:{name}:{partition}.

Experimental: Horizon Dashboard Integration

You can enable experimental Horizon events integration to see completed/recent jobs in the Horizon dashboard:

Or via environment variable:

Warning: This feature is experimental and adds a small overhead per job (writing to Horizon's Redis keys). Test thoroughly in your environment before using in production.

What this enables:

• Completed jobs appear in Horizon dashboard
• Recent jobs list works
• Job metrics (throughput, runtime) are tracked

What still doesn't work:

• Pending jobs count (architectural limitation)
• horizon:clear command (use balanced-queue:clear)

Monitoring Commands

Use built-in commands instead of Horizon for queue management:

Example output of balanced-queue:table --watch:

Configuration

Partition Strategies

Choose how partitions are selected for processing:

Concurrency Limiters

Control how many jobs run simultaneously per partition:

Environment Variables

Partition Keys

Automatic Detection

The BalancedDispatchable trait automatically detects partition key from common property names:

Supported auto-detected properties: $userId, $user_id, $tenantId, $tenant_id

Explicit Partition

Custom Partition Logic

Override getPartitionKey() in your job:

Global Partition Resolver

Set a default resolver in config for all jobs:

Partition Resolution Priority

When determining the partition key, the following order is used:

The first non-null value wins. This allows you to:

• Override everything with onPartition() at dispatch time
• Define custom logic per job with getPartitionKey()
• Set a global default with partition_resolver in config
• Rely on automatic detection for simple cases

Advanced Usage

Programmatic Metrics

Custom Strategy

Register in config:

Custom Limiter

Redis Structure

Understanding the Redis key structure helps with debugging:

Example with default prefix and queue:

Debugging with Redis CLI

Troubleshooting

Jobs not executing

1. Check that connection name matches in queue.php and when dispatching:

2. Verify worker is running with correct connection:

Jobs stuck / not completing

Check for orphaned active job entries:

Horizon shows 0 pending jobs

This is expected behavior. Use balanced-queue:table command instead:

Workers idle but jobs pending

Check if all partitions hit their concurrency limit:

If all partitions show Active = max_concurrent, workers are waiting for slots to free up.

Monitoring with Prometheus & Grafana

The package provides optional HTTP endpoints for monitoring integration.

Enable Monitoring Endpoints

Available Endpoints

Security

By default, endpoints are protected with IP whitelist middleware. Configure allowed IPs in config/balanced-queue.php:

Prometheus Setup

1. Add scrape config to prometheus.yml:

2. Available metrics:

Grafana with Infinity Plugin (Real-time without Prometheus)

For real-time monitoring without Prometheus server, use Grafana Infinity datasource:

1. Install Infinity plugin in Grafana
2. Create datasource pointing to your app

3. Use the JSON endpoint:

4. Configure Infinity datasource query:

   • Type: JSON
   • URL: https://your-app.com/balanced-queue/metrics/json
   • Parser: Backend
5. For queue summary table: Rows/Root: queues
6. For partition details: Rows/Root: queues[0].partitions (or use JSONata for all partitions)

Testing

Requirements

• PHP 8.1+
• Laravel 10.x, 11.x, 12.x or 13.x
• Redis with phpredis or predis
• Laravel Horizon (optional, for worker management)

Credits

Inspired by aloware/fair-queue with improvements:

• Multiple partition strategies (not just random)
• Built-in concurrency limiters
• Artisan commands for monitoring
• Cleaner, extensible architecture

License

MIT License. See LICENSE for details.