Download the PHP package kefyusuf/laravel-shard without Composer

Laravel Shard

A modular shard locator for Laravel applications that provides deterministic shard routing for shard-key-aware Eloquent workflows. Redis-backed maps and shard-aware queues are optional modules.

✨ Features

• Deterministic Shard Routing: Route shard-key-aware reads and writes to the correct shard
• Redis-Based Lookup: Persistent key-to-shard mappings stored in Redis
• Multiple Strategies: Support for Modulo, Consistent Hashing, Range-based, and Virtual Bucket sharding
• Request Routing: Route middleware and shard-aware builders for single-shard operations
• Model Integration: Easy integration using the Shardable trait
• Performance Monitoring: Built-in monitoring and health checks
• Cross-Shard Operations: Support for cross-shard queries and aggregations
• Laravel Integration: Native Laravel service provider with Artisan commands
• Octane Compatible: Locator state is flushed between Octane worker requests automatically
• Laravel Pulse Ready: Shard routing distribution recorded and visualized on the Pulse dashboard when Pulse is installed
• Tenancy Bridge: Use stancl/tenancy or spatie/laravel-multitenancy with the tenant id as shard key (REDIS_SHARD_TENANCY_DRIVER)
• Read Replicas: Map per-shard replica connections — reads go to the replica, writes stay on the shard
• Cross-Shard Transactions: ShardManager::transaction() coordinates begin/commit/rollback across shards (best-effort, documented)
• Write-Fenced Rebalancing: idempotent rebalance with write fencing — a concurrent write can never be lost; virtual buckets make resharding move ~1/N of the keys

📋 Requirements

• PHP 8.2, 8.3, or 8.4
• Laravel 10.x, 11.x, 12.x, or 13.x
• Multiple database connections configured

Optional modules

With redis disabled the package falls back to ArrayShardLocator (process-local map) and deterministic strategy routing. No Redis server is required.

📚 Documentation

• Quick Start Guide - Get started in 5 minutes
• API Reference - Complete API documentation
• Rebalance Operations - Dry-run, gradual moves, idempotent rebalance, write fencing, virtual buckets
• Cross-Shard Transactions - ShardManager::transaction() semantics and reconciliation patterns
• Migration Guide - Migrate existing applications
• Performance Guide - Optimization and benchmarks
• Release Runbook - Package + consumer release checklist
• Example Application - Multi-tenant SaaS example
• Load Testing - Performance testing tools

🚀 Installation

Install the package via Composer:

The package will automatically register its service provider thanks to Laravel's package auto-discovery.

Publish the configuration file:

Run the migrations:

🐳 Local Development (Docker)

The repository ships with a Docker stack that bundles PHP (with the redis extension via PECL), Composer, and a Redis 7 service so you can run the full test suite without installing anything on the host.

Override the PHP version via PHP_VERSION (e.g. make PHP_VERSION=8.4 test).

⚙️ Configuration

After publishing the config file, configure your shards in config/redis_sharding.php:

Important Constraints

• Shard-aware query routing is deterministic only when the builder can resolve a single shard from the shard key, currently via explicit shard connection binding, find, shard-key = predicates, and shard-key whereIn(...) predicates.
• Queries that cannot be routed safely now fail fast instead of silently falling back to the default connection.
• Authoritative key-to-shard mappings are persisted in Redis and no longer expire automatically.
• Auto-increment primary keys require an explicit shard key value, request shard binding, or explicit connection before insert.
• modulo is suitable only for fixed shard topologies. Adding or removing shards with modulo requires planned data migration.

Supported Query Patterns

Fail-Fast Query Patterns

• Queries without an explicit shard connection or shard-key predicate
• Shard-key predicates using unsupported operators such as whereBetween(...)
• Queries that mix shard-key routing with orWhere(...)
• upsert(...) payloads that omit the shard key or contain unresolvable shard-key values

These paths now throw ShardingException instead of silently using the default connection.

Locator Fallback Store

If you want shard lookups to survive a Redis outage beyond the current PHP process, configure redis_sharding.locator.fallback_store with a Laravel cache store that is independent from Redis.

• Recommended: database or file
• Acceptable: any shared non-Redis cache backend with separate failure characteristics
• Not useful for outage isolation: a cache store backed by the same Redis cluster
• Test-only: array, because it is process-local

Example:

Health / metrics endpoint

Enable a JSON health probe for load balancers and uptime checks (no Redis required):

Status is ok (all shards reachable), degraded (partial), or down (none reachable).

Add ?detail=1 for a full diagnostics payload (latency, distribution balance, modules, issues), or use the console:

Programmatic access:

🎯 Quick Start

The steps below match the package consumer smoke test (examples/smoke.php), which installs this package into a fresh Laravel app on every CI run.

1. Install

2. Enable only the modules you need

With redis disabled the package uses ArrayShardLocator and deterministic strategy routing — no Redis server required.

3. Add sharding to your models

Use the Shardable trait and override getShardKeyName() (do not redeclare $shardKey; the trait already defines it):

4. Use shard-aware queries

5. Dispatch shard-aware jobs (queue module)

RestoreShardContext job middleware rebinds the shard connection on the worker before handle() runs.

6. Use middleware for request routing

7. Verify the wiring

🔧 Sharding Strategies

The package supports four different sharding strategies:

1. Modulo Strategy

Simple modulo-based distribution. Use only when the shard list is effectively static.

Pros: Simple, predictable distribution Cons: Adding/removing shards remaps most keys and requires explicit data migration

2. Consistent Hashing Strategy

Uses consistent hashing algorithm for better distribution when shards are added/removed.

Pros: Minimal data movement when scaling, good distribution Cons: A rehash still displaces ~half the keys; resharding is all-or-nothing per run

3. Range-Based Strategy

Distributes data based on key ranges.

Pros: Good for time-series or sequential data Cons: Can create hotspots if data isn't evenly distributed

4. Virtual Bucket Strategy (Recommended for resharding)

A key hashes into a fixed bucket (crc32 % N, default 1024) and a persistent bucket→shard map decides placement. Buckets never move implicitly — you move them explicitly and rebalance carries only that bucket's keys (~1/N of the data per move).

See docs/REBALANCE.md for the full workflow.

🎛️ Artisan Commands

Shard Management

Virtual Bucket Management

Monitoring & Analysis

Health Monitoring

Performance Analysis

JSON Output Quick Reference

All commands that support output formatting only accept --format=table or --format=json.

Common payload conventions:

• summary.status: ok, error, or command-specific lifecycle value (dry_run, completed, etc.)
• error: Present when command fails in JSON mode
• Detail blocks by command:
  • install: summary.published_config, summary.ran_migrations, summary.skipped_migrations
  • create: summary.shard, summary.driver, summary.database, summary.created
  • status: summary + shards or tables detail arrays depending on options
  • cleanup: summary + report
  • health: summary + issues (+ fix when --fix is used)
  • analyze: summary + results/analysis
  • rebalance: summary (+ moves in dry-run mode)

📊 Monitoring & Performance

Built-in Monitoring System

The package includes a comprehensive monitoring system that tracks shard health, performance, and distribution:

Advanced Health Monitoring

Caching & Performance Optimization

🔄 Advanced Usage

Cross-Shard Query Builder

The package now includes a powerful cross-shard query builder that allows you to query across all shards seamlessly:

Convenient Cross-Shard Methods

Batch Operations Across Shards

Manual Shard Selection

🛡️ Configuration Validation

The package includes comprehensive configuration validation to prevent common setup issues:

Configuration Features

• Connection Validation: Ensures all shard connections are properly configured
• Strategy Validation: Validates sharding strategy classes and interfaces
• Redis Validation: Checks Redis connection configuration
• Auto-Provisioning Validation: Validates auto-scaling settings
• Helpful Recommendations: Provides optimization suggestions

🧪 Comprehensive Testing

The package includes extensive testing coverage:

Test Suites

Test Categories

Unit Tests:

• ModuloStrategyTest - Tests modulo distribution algorithm
• ConsistentHashingStrategyTest - Tests consistent hashing with shard changes
• RangeBasedStrategyTest - Tests range-based distribution

Integration Tests:

• ShardManagerTest - Tests shard management functionality
• ShardLocatorTest - Tests Redis-based shard location
• Cross-component interaction testing

Feature Tests:

• CreateShardCommandTest - Tests shard creation command
• End-to-end workflow testing
• Command validation and error handling

Testing Your Implementation

🏗️ Real-World Examples

E-commerce Application

Multi-Tenant SaaS Application

Social Media Platform

📋 Best Practices

1. Choosing the Right Shard Key

2. Monitoring and Maintenance

3. Handling Relationships

4. Performance Optimization

🤝 Contributing

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Development Setup

🔧 Troubleshooting

Common Issues

1. Configuration Validation Errors

2. Redis Connection Issues

3. Shard Distribution Problems

4. Performance Issues

Debug Mode

Performance Tuning

📚 Additional Resources

• Examples Directory - Complete working examples
• Test Suite - Comprehensive test coverage
• Configuration Reference - Full configuration options
• API Documentation - Detailed API reference

🆕 Changelog

See CHANGELOG.md for the full release history (latest: v5.0.0 — virtual bucket sharding, cross-shard transactions, deep-module refactor, read replicas, write fencing, Pulse/Octane/tenancy integrations).

📝 License

This package is open-sourced software licensed under the MIT license.

🙏 Acknowledgments

• Laravel Framework for the excellent foundation
• Redis for fast key-value storage
• The PHP community for continuous inspiration

Made for the Laravel community.

For questions, issues, or contributions, please visit our GitHub repository.