Download the PHP package sanmai/pipeline without Composer

sanmai/pipeline provides a fluent, memory-efficient way to process iterable data in PHP. It uses lazy evaluation with generators, allowing you to build complex data processing pipelines that are easy to read and write.

Inspired by the pipe operator (|) in shells, it lets you chain operations like map, filter, reduce, zip, and more. Because it processes items one by one and only when needed, it's highly effective for large or even infinite collections without exhausting memory.

The library is rigorously tested and robust. Pipeline neither defines nor throws any exceptions.

Install

The latest version requires PHP 8.2 or above.

Some earlier versions work under PHP 5.6 and above, but they are not as feature-complete.

Documentation

Read the full documentation

The documentation includes:

• Quick start guide to get up and running
• Full Pipeline API reference for a deep dive
• Cookbook with practical recipes and patterns
• Best practices with tips for effective usage

Use

API entry points

All entry points always return an instance of the pipeline.

Instance methods in a nutshell

Pipeline implements IteratorAggregate and can be used as any other iterable.

Pipeline can be used as an argument to count(). Implements Countable. Be warned that operation of counting values is a terminal operation.

In general, Pipeline instances are mutable, meaning every Pipeline-returning method returns the very same Pipeline instance. This gives us great flexibility on trusting someone or something to add processing stages to a Pipeline instance, while also avoiding non-obvious mistakes, raised from a need to strictly follow a fluid interface. E.g. if you add a processing stage, it stays there no matter if you capture the return value or not. This peculiarity could have been a thread-safety hazard in other circumstances, but under PHP this is not an issue.

Caveats

• Since most callback are lazily evaluated as more data coming in and out, you must consume the results with a plain foreach or use a reduce() to make sure processing happens.

  Almost nothing will happen unless you use the results. That's the point of lazy evaluation after all!

• That said, if a non-generator used to seed the pipeline, it will be executed eagerly.

  In the above case the pipeline will store an array internally, with which the pipeline will operate eagerly whenever possible. Ergo, when in doubt, use a generator.

• Keys for yielded values are being kept as is on a best effort basis, so one must take care when using iterator_to_array() on a pipeline: values with duplicate keys will be discarded with only the last value for a given key being returned.

  Safer would be to use provided toList() method. It will return all values regardless of keys used, making sure to discard all keys in the process.

  If necessary to preserve the keys, there's a sister method toAssoc().

• The resulting pipeline is an iterator and should be assumed not rewindable, just like generators it uses.

  Although there are some cases where a pipeline can be rewound and reused just like a regular array, if you need to pause iteration and continue later, or if you need to iterate strictly once without accidental resets or exceptions, use $pipeline->cursor().

• Pipeline implements IteratorAggregate which is not the same as Iterator. Where the latter needed, the pipeline can be wrapped with an IteratorIterator:

• Iterating over a pipeline all over again results in undefined behavior. Best to avoid doing this. If you need to break out of iteration and continue later, see cursor().

Classes and interfaces: overview

• \Pipeline\Standard is the main user-facing class for the pipeline with sane defaults for most methods.

This library is built to last. There's not a single place where an exception is thrown. Never mind any asserts whatsoever.

Methods

__construct()

Takes an instance of Traversable or none. In the latter case the pipeline must be primed by passing an initial generator to the map method.

$pipeline->map()

Takes a processing stage in a form of a generator function or a plain mapping function. Provided no callback does nothing.

Can also take an initial generator, where it must not require any arguments.

$pipeline->flatten()

Flatten inputs:

$pipeline->unpack()

An extra variant of map which unpacks arrays into arguments for a callback.

Where with map() you would use:

With unpack() these things are done behind the scene for you:

You can have all kinds of standard type checks with ease too.

With no callback, the default callback for unpack() will flatten inputs as in flatten().

$pipeline->cast()

Works similarly to map, but does not have a special treatment for generators. Think of array_map.

For this example, where map() would have filled the pipeline with a series of payments, cast() will add a generator for each customer.

$pipeline->zip()

Sequence-joins several iterables together, forming a feed with elements side by side:

With iterators with unequal number of elements, missing elements are left as nulls.

$pipeline->select()

Selects elements for which the callback returns true. filter() is an alias with strict: false default.

By default, select() only removes null and false values.

With the optional strict: false parameter, it removes all falsy values like array_filter:

The optional onReject callback allows side effects (like logging) for rejected items. It receives ($value, $key) like tap():

$pipeline->slice()

Takes offset and length arguments, functioning in a very similar fashion to how array_slice does with $preserve_keys set to true.

This example will remove first and last elements of the sequence.

Implementation uses a rolling window buffer for negative values of offset and length, and falls back on plain old array_slice for input arrays.

$pipeline->reduce()

Takes a reducing callback not unlike that of array_reduce with two arguments for the value of the previous iteration and for the current item. As a second argument it can take an initial value.

The pipeline has a default callback that sums all values.

$pipeline->toList()

Returns an array with all values from a pipeline. All array keys are ignored to make sure every single value is returned.

If in the example about one would use iterator_to_array($result) they would get just [3, 4].

$pipeline->tap()

Performs side effects on each element without changing the values in the pipeline. Useful for debugging, logging, or other side effects.

The tap() method executes the callback for each element as it flows through the pipeline, but the original values continue unchanged to the next stage.

$pipeline->each()

Eagerly iterates over the sequence using the provided callback.

Discards the sequence after iteration unless instructed otherwise by the second argument.

$pipeline->getIterator()

A method to conform to the Traversable interface. In case of unprimed \Pipeline\Standard it'll return an empty array iterator, essentially a no-op pipeline. Therefore this should work without errors:

This allows to skip type checks for return values if one has no results to return: instead of false or null it is safe to return an unprimed pipeline.

$pipeline->cursor()

Returns a forward-only iterator that maintains position across iterations. A cursor allows breaking out of a loop and continuing later:

$pipeline->runningVariance()

Computes online statistics for the sequence: counts, sample mean, sample variance, standard deviation. You can access these numbers on the fly with methods such as getCount(), getMean(), getVariance(), getStandardDeviation().

This method also accepts an optional cast callback that should return float|null: null values are discarded. Therefore you can have several running variances computing numbers for different parts of the data.

As you process orders, you will be able to access $varianceForShippedOrders->getMean() and $varianceForPaidOrders->getMean().

This computation uses Welford's online algorithm, therefore it can handle very large numbers of data points.

$pipeline->finalVariance()

A convenience method to computes the final statistics for the sequence. Accepts an optional cast method, else assumes the sequence contains valid numbers.

Type Safety

The library provides extensive generic type tracking, whether you prefer method chaining or not.

Given the following type:

A static analyzer will correctly infer that these examples are sound:

But if you were to change the signature of the class:

PHPStan will correctly note that:

• The first parameter of class Foo constructor expects string but int given.
• There is a call to an undefined method Foo::bar().

Contributions

Contributions to documentation and test cases are welcome. Bug reports are welcome too.

API is expected to stay as simple as it is, though.

About collection pipelines in general

About collection pipelines programming pattern by Martin Fowler.

In a more general sense this library implements a subset of CSP paradigm, as opposed to Actor model.

What else is out there:

• Pipe operator from Hack is about same, only won't work for generators, and not under the regular PHP. See a proposal for a similar operator for JavaScript.
• nikic/iter provides functions like array_map and such, but returning lazy generators. You'll need quite some glue to accomplish the same thing Pipeline does out of box, not to mention some missing features.
• League\Pipeline is good for single values only. Similar name, but very different purpose. Not supposed to work with sequences of values. Each stage may return only one value.
• Illuminate\Support\Collection a fluent wrapper for working with arrays of data. Can only work with arrays, also immutable, which is kind of expected for an array-only wrapper.
• Knapsack is a close call. Can take a Traversable as an input, has lazy evaluation. But can't have multiple values produced from a single input. Has lots of utility functions for those who need them: they're out of scope for this project.
• transducers.php is worth a close look if you're already familiar transducers from Clojure. API is not very PHP-esque. Read as not super friendly. Detailed write-up from the author.
• Primitives for functional programming in PHP by Lars Strojny et al. is supposed to complement currently existing PHP functions, which it does, although it is subject to some of the same shortcomings as are array_map and array_filter. No method chaining.
• Chain provides a consistent and chainable way to work with arrays in PHP, although for arrays only. No lazy evaluation.
• Simple pipes with PHP generators by Hugh Grigg. Rationale and explanation for an exceptionally close concept. Probably one can use this library as a drop-in replacement, short of different method names.
• loophp's Collection looks like a viable alternative to this library, as far as processing of multi-gigabyte log files goes. Supports fluent interface. It takes the immutability as a first principle, even though PHP's generators are inherently mutable.
• If you're familiar with Java, package java.util.stream offers an implementation of the same concept.
• Collection types in Scala.

Submit a PR to add yours.

More badges