Download the PHP package spiritinlife/php-mapreduce without Composer

On this page you can find all versions of the php package spiritinlife/php-mapreduce. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.

FAQ

After the download, you have to make one include require_once('vendor/autoload.php');. After that you have to import the classes with use statements.

Example:
If you use only one package a project is not needed. But if you use more then one package, without a project it is not possible to import the classes with use statements.

In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
Some PHP packages are not free to download and because of that hosted in private repositories. In this case some credentials are needed to access such packages. Please use the auth.json textarea to insert credentials, if a package is coming from a private repository. You can look here for more information.

  • Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
  • To use Composer is sometimes complicated. Especially for beginners.
  • Composer needs much resources. Sometimes they are not available on a simple webspace.
  • If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
  • Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Please rate this library. Is it a good library?

Informations about the package php-mapreduce

PHP MapReduce

A high-performance, framework-agnostic MapReduce implementation for PHP that processes large datasets using true parallel processing with separate PHP worker processes and disk-based storage.

PHP Version

Why Use This?

Installation

Requirements: PHP 8.1+

Quick Start

Word count example:

How It Works

MapReduce processes data in three parallel phases using separate PHP worker processes:

  1. Map Phase - Input is streamed in batches to parallel worker processes. Each worker emits key-value pairs to partitioned temporary files.
  2. Shuffle Phase - Partitions are sorted and grouped in parallel using memory-efficient external sorting with separate worker processes.
  3. Reduce Phase - Partitions are processed in parallel by worker processes, aggregating values by key.

Each phase uses spatie/async to spawn separate PHP processes, enabling true CPU parallelism for compute-intensive operations.

⚠️ Important: Because mapper and reducer functions run in separate processes, you cannot use closure use clauses to access external variables. Use the context() method instead. See Parallel Processing & Context for details.

Input Types

MapReduce accepts any iterable as input - arrays, generators, iterators, or custom iterables.

💡 True Streaming: Large files, database cursors, and API paginations are consumed as they're produced - never loading the entire dataset into memory.

Arrays and Basic Iterables

Files and Streams

CSV Files

JSON Lines (JSONL)

Database Results

Directory Scanning

API Reference

Core Methods (Required)

input(iterable $input)

Set the input data - any array or iterable.

map(callable $mapper)

Define the transformation function. Receives ($value, $context) and must yield [$key, $value] pairs.

reduce(callable $reducer)

Define the aggregation function using an accumulator + iterator pattern.

The reducer receives ($key, $valuesIterator, $context) where:

You manually accumulate results by iterating over the values. This approach never loads all values for a key into memory at once, which is critical for keys with many values (e.g., a global counter with millions of entries).

context(mixed $data)

Pass shared data to mapper and reducer functions executing in parallel processes.

Because mapper and reducer functions run in separate PHP processes, you cannot use closure use clauses to access external variables. Use context() instead to pass serializable data that will be available to all workers.

The context is passed as an additional parameter to both mapper and reducer:

See Parallel Processing & Context for details.

execute(): Generator

Run the job and return results as a generator

Configuration Methods (Optional)

concurrent(int $workers)

Number of parallel worker processes. Default: 4

Tip: Match your CPU core count. For I/O-bound tasks, you can use 2-3x cores.

partitions(int $count)

Number of reduce partitions for parallel processing. Default: same as concurrent()

Tip: Use 1-2x concurrency level. More partitions = better parallelism but more overhead.

mapperBatchSize(int $items)

Controls: Items sent to each parallel worker process. Default: 500

Determines how many input items are batched together and sent to a worker process. This is critical for memory management because it affects how many tasks the parent process creates.

Tradeoffs:

Guidelines:

⚠️ Critical: Batch Size vs Memory

Parent process memory is proportional to the number of tasks, not dataset size:

  • 1M records ÷ 500 batch = 2,000 tasks (may use 100MB+ parent memory)
  • 1M records ÷ 5,000 batch = 200 tasks (uses ~10MB parent memory)

For large datasets (>100K records), use larger batch sizes to reduce parent process memory overhead.

shuffleChunkSize(int $records)

Controls: Memory usage during shuffle phase external sorting. Default: 10000

Determines how many records accumulate in memory before being sorted and written as a chunk file during external sorting. Only affects the shuffle phase.

Memory impact: shuffleChunkSize × average_record_size bytes per sort operation

Guidelines:

bufferSize(int $records)

Controls: File I/O write buffering across all phases. Default: 1000

Determines how many records are buffered in memory before flushing to disk. Affects the frequency of fwrite() system calls. Used by map, shuffle, and reduce phases.

Memory impact: bufferSize × average_record_size × num_partitions bytes in map phase (each partition has its own buffer)

Guidelines:

💡 Three Independent Parameters:

  • mapperBatchSize - Controls worker granularity & parent memory (map phase only)
  • shuffleChunkSize - Controls sorting memory (shuffle phase only)
  • bufferSize - Controls write batching (all phases)

These handle different concerns:

  • Large mapperBatchSize reduces process spawning overhead AND parent memory (critical for >100K records)
  • Large shuffleChunkSize creates fewer chunk files to merge
  • Large bufferSize reduces system call overhead

Most impactful for large datasets: Increase mapperBatchSize first!

workingDirectory(string $path)

Directory for temporary files. Default: system temp

Tip: Use SSD storage for better performance on large datasets.

autoload(string $path)

Configure autoloader path for parallel worker processes. Default: none

When using parallel processing, worker processes spawn in separate contexts and may need to load dependencies. This method specifies the path to your autoloader (typically vendor/autoload.php).

When to use:

Note: Most of the time you won't need this - the library handles common cases automatically. Only use if you encounter class loading issues in parallel workers.

partitionBy(callable $partitioner)

Custom function to control which partition a key goes to. Receives ($key, $numPartitions, $context) and must return 0 to partitions-1.

Examples

Aggregating Sales Data

Building an Inverted Index

Log Analysis

Using Context for Shared Data

Parallel Processing & Context

Important: Closure use Clauses Don't Work

Because MapReduce uses true parallel processing with separate PHP processes (via spatie/async), variables captured with use clauses are NOT available in mapper and reducer functions.

Solution: Use context() Method

Pass data via the context() method, which serializes and provides it to all workers:

Context Guidelines

What can be passed as context:

Best practices:

Real-world example (collaborative filtering):

Performance Tuning

Quick Reference

Memory vs Performance

Low Memory System (2GB RAM):

High Performance System (SSD, 32GB RAM):

Tuning independently:

Benchmarks

Tested on Apple M2 Pro (10-core), 16GB RAM, SSD with spatie/async parallel processing:

Records Concurrency Time Parent Memory* Batch Size
10K 4 0.3s 6MB 500
100K 4 1.2s 0MB 2,000
1M 8 7.9s 8MB 5,000
10M 8 1m23s 10MB 5,000

*Parent process memory only - worker processes use separate memory space

Key Insight: Using larger mapperBatchSize for datasets >100K dramatically reduces parent memory (from 100MB+ to <10MB) by creating fewer tasks. See the critical callout in mapperBatchSize() documentation above.

Run your own benchmarks:

Testing

Run tests:

Limitations

License

MIT License - see LICENSE file


All versions of php-mapreduce with dependencies

PHP Build Version
Package Version
Requires php Version ^8.1
ext-json Version *
spatie/async Version ^1.6
Composer command for our command line client (download client) This client runs in each environment. You don't need a specific PHP version etc. The first 20 API calls are free. Standard composer command

The package spiritinlife/php-mapreduce contains the following files

Loading the files please wait ...