Download the PHP package spiritinlife/php-mapreduce without Composer
On this page you can find all versions of the php package spiritinlife/php-mapreduce. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Download spiritinlife/php-mapreduce
More information about spiritinlife/php-mapreduce
Files in spiritinlife/php-mapreduce
Package php-mapreduce
Short Description A high-performance, framework-agnostic MapReduce implementation for PHP using Spatie Async
License MIT
Informations about the package php-mapreduce
PHP MapReduce
A high-performance, framework-agnostic MapReduce implementation for PHP that processes large datasets using true parallel processing with separate PHP worker processes and disk-based storage.
Why Use This?
- True CPU parallelism - spawns separate PHP processes for parallel execution, not async coroutines
- Process datasets larger than RAM using memory-efficient disk storage
- Stream iterators without loading into memory - process large files, database cursors, and API responses efficiently
- Utilize multiple CPU cores for CPU-intensive transformations
- Framework-agnostic - works with any PHP project
- Handle millions of records with predictable memory usage
Installation
Requirements: PHP 8.1+
Quick Start
Word count example:
How It Works
MapReduce processes data in three parallel phases using separate PHP worker processes:
- Map Phase - Input is streamed in batches to parallel worker processes. Each worker emits key-value pairs to partitioned temporary files.
- Shuffle Phase - Partitions are sorted and grouped in parallel using memory-efficient external sorting with separate worker processes.
- Reduce Phase - Partitions are processed in parallel by worker processes, aggregating values by key.
Each phase uses spatie/async to spawn separate PHP processes, enabling true CPU parallelism for compute-intensive operations.
⚠️ Important: Because mapper and reducer functions run in separate processes, you cannot use closure
useclauses to access external variables. Use thecontext()method instead. See Parallel Processing & Context for details.
Input Types
MapReduce accepts any iterable as input - arrays, generators, iterators, or custom iterables.
💡 True Streaming: Large files, database cursors, and API paginations are consumed as they're produced - never loading the entire dataset into memory.
Arrays and Basic Iterables
Files and Streams
CSV Files
JSON Lines (JSONL)
Database Results
Directory Scanning
API Reference
Core Methods (Required)
input(iterable $input)
Set the input data - any array or iterable.
map(callable $mapper)
Define the transformation function. Receives ($value, $context) and must yield [$key, $value] pairs.
reduce(callable $reducer)
Define the aggregation function using an accumulator + iterator pattern.
The reducer receives ($key, $valuesIterator, $context) where:
$key- The current key being reduced$valuesIterator- Iterator that streams values one at a time for memory efficiency$context- Optional shared context data (see context() method)
You manually accumulate results by iterating over the values. This approach never loads all values for a key into memory at once, which is critical for keys with many values (e.g., a global counter with millions of entries).
context(mixed $data)
Pass shared data to mapper and reducer functions executing in parallel processes.
Because mapper and reducer functions run in separate PHP processes, you cannot use closure use clauses to access external variables. Use context() instead to pass serializable data that will be available to all workers.
The context is passed as an additional parameter to both mapper and reducer:
- Mapper:
function($value, $context) - Reducer:
function($key, $valuesIterator, $context)
See Parallel Processing & Context for details.
execute(): Generator
Run the job and return results as a generator
Configuration Methods (Optional)
concurrent(int $workers)
Number of parallel worker processes. Default: 4
Tip: Match your CPU core count. For I/O-bound tasks, you can use 2-3x cores.
partitions(int $count)
Number of reduce partitions for parallel processing. Default: same as concurrent()
Tip: Use 1-2x concurrency level. More partitions = better parallelism but more overhead.
mapperBatchSize(int $items)
Controls: Items sent to each parallel worker process. Default: 500
Determines how many input items are batched together and sent to a worker process. This is critical for memory management because it affects how many tasks the parent process creates.
Tradeoffs:
- Smaller batches = Better load balancing, MORE tasks = MORE parent memory
- Larger batches = Less overhead, fewer tasks, LESS parent memory
Guidelines:
- Small datasets (<10K records): 100-500 (default)
- Medium datasets (10K-100K): 500-2,000
- Large datasets (100K-1M): 2,000-5,000
- Very large datasets (>1M): 5,000-10,000
⚠️ Critical: Batch Size vs Memory
Parent process memory is proportional to the number of tasks, not dataset size:
- 1M records ÷ 500 batch = 2,000 tasks (may use 100MB+ parent memory)
- 1M records ÷ 5,000 batch = 200 tasks (uses ~10MB parent memory)
For large datasets (>100K records), use larger batch sizes to reduce parent process memory overhead.
shuffleChunkSize(int $records)
Controls: Memory usage during shuffle phase external sorting. Default: 10000
Determines how many records accumulate in memory before being sorted and written as a chunk file during external sorting. Only affects the shuffle phase.
Memory impact: shuffleChunkSize × average_record_size bytes per sort operation
Guidelines:
- Small datasets (<100K): 5,000-10,000
- Medium datasets (100K-1M): 10,000-50,000 (default: 10,000)
- Large datasets (>1M): 50,000-100,000
bufferSize(int $records)
Controls: File I/O write buffering across all phases. Default: 1000
Determines how many records are buffered in memory before flushing to disk. Affects the frequency of fwrite() system calls. Used by map, shuffle, and reduce phases.
Memory impact: bufferSize × average_record_size × num_partitions bytes in map phase (each partition has its own buffer)
Guidelines:
- Small datasets (<10K): 100-500
- Medium datasets (10K-1M): 1000-5000 (default: 1000)
- Large datasets (>1M): 5000-10000
💡 Three Independent Parameters:
mapperBatchSize- Controls worker granularity & parent memory (map phase only)shuffleChunkSize- Controls sorting memory (shuffle phase only)bufferSize- Controls write batching (all phases)These handle different concerns:
- Large
mapperBatchSizereduces process spawning overhead AND parent memory (critical for >100K records)- Large
shuffleChunkSizecreates fewer chunk files to merge- Large
bufferSizereduces system call overheadMost impactful for large datasets: Increase
mapperBatchSizefirst!
workingDirectory(string $path)
Directory for temporary files. Default: system temp
Tip: Use SSD storage for better performance on large datasets.
autoload(string $path)
Configure autoloader path for parallel worker processes. Default: none
When using parallel processing, worker processes spawn in separate contexts and may need to load dependencies. This method specifies the path to your autoloader (typically vendor/autoload.php).
When to use:
- When mapper/reducer functions use classes from external libraries
- When using custom classes that need autoloading
- If you get "class not found" errors in worker processes
Note: Most of the time you won't need this - the library handles common cases automatically. Only use if you encounter class loading issues in parallel workers.
partitionBy(callable $partitioner)
Custom function to control which partition a key goes to. Receives ($key, $numPartitions, $context) and must return 0 to partitions-1.
Examples
Aggregating Sales Data
Building an Inverted Index
Log Analysis
Using Context for Shared Data
Parallel Processing & Context
Important: Closure use Clauses Don't Work
Because MapReduce uses true parallel processing with separate PHP processes (via spatie/async), variables captured with use clauses are NOT available in mapper and reducer functions.
Solution: Use context() Method
Pass data via the context() method, which serializes and provides it to all workers:
Context Guidelines
What can be passed as context:
- Arrays, scalars (int, float, string, bool)
- Objects (as long as they're serializable - no resources, DB connections, or file handles)
- Nested data structures
- Class constants and configuration values
Best practices:
- Keep context reasonably sized (< 100MB recommended) - it's serialized to each worker
- Context is read-only - changes in one worker won't affect others
- All data must be serializable (no resources, closures, or database connections)
Real-world example (collaborative filtering):
Performance Tuning
Quick Reference
Memory vs Performance
Low Memory System (2GB RAM):
High Performance System (SSD, 32GB RAM):
Tuning independently:
- Large dataset (>100K records)? CRITICAL: Increase
mapperBatchSizeto drastically reduce parent process memory (can drop from 100MB+ to <10MB) - Slow disk? Increase
bufferSizeto batch more writes - Limited RAM during shuffle? Decrease
shuffleChunkSizeto reduce sort memory - Many concurrent workers + partitions? Decrease
bufferSize(memory = workers × partitions × bufferSize) - Parent process using too much memory? Increase
mapperBatchSize(fewer tasks = less overhead)
Benchmarks
Tested on Apple M2 Pro (10-core), 16GB RAM, SSD with spatie/async parallel processing:
| Records | Concurrency | Time | Parent Memory* | Batch Size |
|---|---|---|---|---|
| 10K | 4 | 0.3s | 6MB | 500 |
| 100K | 4 | 1.2s | 0MB | 2,000 |
| 1M | 8 | 7.9s | 8MB | 5,000 |
| 10M | 8 | 1m23s | 10MB | 5,000 |
*Parent process memory only - worker processes use separate memory space
Key Insight: Using larger mapperBatchSize for datasets >100K dramatically reduces parent memory (from 100MB+ to <10MB) by creating fewer tasks. See the critical callout in mapperBatchSize() documentation above.
Run your own benchmarks:
Testing
Run tests:
Limitations
- Single machine only (not a distributed cluster)
- Requires disk space for intermediate files
- Process spawning overhead makes it inefficient for tiny datasets (<1000 records)
- Closure
useclauses don't work in mapper/reducer functions due to parallel processing - usecontext()method instead (see Parallel Processing & Context)
License
MIT License - see LICENSE file