Download the PHP package zenstruck/collection without Composer

zenstruck/collection

A Collection interface for iterating, paginating and filtering any set of data - arrays, iterators, doctrine/collections instances and Doctrine ORM queries - all behind the same API:

It really shines with Doctrine ORM, where a custom EntityResult object makes everything lazy by default - no more loading huge amounts of entities into memory at once:

An EntityResult is an unexecuted query you can pass around and derive from. It is lazy and immutable, so each of these runs its own optimized query:

Features:

1. Collections: A lazy, countable, paginatable Collection interface with implementations for arrays, iterables and callables.
2. Pagination: Paginate any collection, with page metadata and lazy page-by-page iteration.
3. Doctrine:
   1. ORM: Lazy, specification-driven repositories with DTO/scalar hydration.
   2. Batch Processing: Memory-safe iteration/mutation of large result sets.
   3. Collection Bridge: Filter and paginate doctrine/collections instances - including relations - without initializing them.
   4. Specifications: Express filters/sorts as objects that are converted into native queries.
4. Symfony Integration: Autowire a lazy-first repository for any entity.
5. Static Analysis: Fully generic - PHPStan knows what your collections contain.
6. Recipes: Worked examples for things that aren't a database.

Installation

The core Collection API has no dependencies. The following packages unlock the optional integrations:

Collections

Zenstruck\Collection is the interface everything in this package implements. It is an IteratorAggregate and Countable that adds transformation (filter(), map(), keyBy()), reduction (first(), find(), reduce()) and pagination - and, where the implementation allows, does all of it lazily.

Creating

The Zenstruck\collect() function wraps any source in the most appropriate implementation:

Each implementation can also be constructed directly if you want a specific one.

The API

Filtering

filter() and find() take a callable(V,K):bool:

Doctrine-backed collections additionally accept specification objects, which are converted into a real query instead of filtering in PHP.

[!NOTE] Specifications are only understood by Doctrine-backed collections. Passing one to ArrayCollection, LazyCollection or any other in-memory implementation throws Zenstruck\Collection\Exception\InvalidSpecification.

Lazy vs Eager

LazyCollection (and the Doctrine implementations) do no work until iterated. Transformations stay lazy - they wrap the source rather than run it:

[!IMPORTANT] "Runs the source" is not the same as "loads the source". Only eager() (and ArrayCollection, which is array-backed to begin with) holds the whole collection in memory - everything else streams one item at a time. That's also why eager() is useful: it lets you iterate repeatedly without re-running an expensive source.

[!WARNING] Generators can't be rewound, so LazyCollection rejects them outright - wrap in a closure instead:

A closure returning an array/Traversable is only executed once and cached; a closure returning a generator is re-executed every time the collection is iterated.

[!TIP] count() on a source that isn't Countable has to iterate all of it. If you have a cheaper way to count, use CallbackCollection.

ArrayCollection

An eager, immutable, array-backed implementation with a much larger API. "Mutations" (set(), unset(), push()) return a new instance:

Named constructors:

In addition to the Collection API:

map() and filter() preserve keys. keyBy() and groupBy() accept Stringable keys and cast them to string.

LazyCollection

Wraps a Traversable or a callable(): iterable. This is what you want for anything expensive - a generator over a large file, an HTTP paginator, a database cursor:

Composing Collections

[!NOTE] FactoryCollection decorates another Collection, so wrap plain iterables in collect() first.

[!WARNING] When preserving keys with ChainCollection, duplicate keys across the inner collections will overwrite each other if the result is converted to an array (ie via eager()).

Pagination

Any collection can be paginated with paginate(), which returns a Page - an iterable of just that page's items, plus the metadata you need to render a pager:

What it costs depends on how much of that metadata you use:

There's only one Page - you don't choose a mode up front, you get one by what your template asks for.

[!NOTE] "Counts the collection" means calling count() on the source, and what that costs is up to the source: it's free for an array, a SELECT COUNT(...) for Doctrine, and a full iteration for a generator or an API-backed Lazy vs Eager.

Simple

Everything a previous/next pager needs comes from the page itself, without counting anything:

Note the 21 items for a page of 20. Knowing whether another page exists doesn't require counting the collection - the page reads one item more than fits on it, and the presence of that extra item is the answer. It's dropped before you see the items, which is why count($page) is still 20.

That one item is often the difference between a bounded amount of work and an unbounded one. Paging an API-backed collection this way reads a single page's worth of results; asking it for a total would walk every page the API has - the recipes work through exactly that, with request counts.

The items are fetched once and cached, so iterating the same Page more than once won't re-run the source.

[!NOTE] Ask for a page past the end and you get an empty one: hasMorePages() and nextPage() report nothing follows, previousPage() still works. See Strict Mode to fall back to the last page instead.

Full

Rendering "Page 2 of 7", numbered links or a "Last" link needs the total, so these count the collection - the first time you ask, and once:

[!NOTE] Out of range arguments are normalized rather than rejected: a page less than 1 becomes 1 and a limit less than 1 becomes the default (20).

Strict Mode

A page number from a bookmark or a hand-edited URL can point past the end of the collection - and a filter that shrank the result set can do the same to a page number that used to be valid. Strict mode falls back to the last page when that happens:

This is free here: the total is already known, so clamping costs nothing extra. The fallback only re-fetches when the requested page really was out of range.

[!TIP] strict() works on a Simple page too, but it gives up the no-counting guarantee: clamping needs the total. An in-range page still counts nothing, but an out of range one reads the empty page, counts, then reads the last page. Keep strict() to Full pagers if that matters.

Templating

Two pager templates are bundled, matching the two modes. Both take the Page and link to the current route, keeping whatever query parameters are already there:

Neither renders anything at all when the collection fits on one page. The markup is unstyled and deliberately plain - style the classes it emits, or copy the template into your app if you want to change the markup itself:

Both accept the same options:

[!NOTE] These require Twig and Symfony's routing (path()), and are registered by the bundle.

Styling

The class names are the styling hooks: pager on both, plus pager-simple/pager-full, and active and disabled on the individual items. With Tailwind that's a few @apply rules and no template to maintain:

Iterating Pages

pages() returns a Pages object - a lazy, page-by-page view of the entire collection. Each page is fetched on its own, so nothing ever holds more than one page in memory:

[!NOTE] count() on Pages is the number of pages, while count() on a Page is the number of items on that page. Use Page::totalCount() for the total number of items.

[!WARNING] Fetching each page independently is only cheap if the source can jump straight to an offset. That's one query per page for Doctrine, but a source that reaches an offset by skipping - a generator, an API - re-reads everything before each page: 2,000 items in pages of 100 costs 249 reads instead of 20. Just iterate the collection for that, or teach it to window itself (recipe).

Pagerfanta

If you'd rather render pagers with Pagerfanta, any collection can be adapted:

Doctrine

ORM

EntityResult

EntityResult is a Collection that wraps a query builder. Nothing is executed until you ask for something, and every method that narrows or transforms it returns a new instance - the original is reusable.

EntityResultQueryBuilder extends Doctrine's QueryBuilder, so everything you already know still works. It adds:

[!WARNING] Iterating an EntityResult streams the rows one at a time but never clears the entity manager, so every entity it hydrates stays in memory. For large result sets, use batchIterate()/batchProcess() instead.

Executing

An EntityResult is a query definition. It runs when you ask it for something, and each of these runs its own query, tailored to what you asked:

This is where pagination's counting shows up as real queries. A previous/next pager is a single query; adding a total makes it two:

Because it's immutable, deriving is free and the original stays usable:

Hydration

By default you get entities back - what the query selects, hydrated the way Doctrine normally would. These methods return a new EntityResult that hydrates each row differently instead:

as() is the general case: it hands you each row and uses whatever you return. What a "row" is depends on what the query selects and which of the above you combined it with - an entity, an array, or a scalar:

The modifier applies everywhere the result produces values - iteration, first(), take(), eager(), paginate() and batch processing all give you PostDto objects.

[!WARNING] There is only one modifier slot: as() replaces anything already set, including the casts behind asInt()/asFloat()/asString() and the wrapper behind withAggregates(). Combine as() with asArray() or a field-selecting query, not with those.

Single Values

A query that selects one aggregate value works the same way - ask for the scalar and take the first row:

Write Queries

EntityResultQueryBuilder is a query builder like any other, so it can also carry a DELETE or an UPDATE. first() executes it and returns the number of affected rows:

[!WARNING] A write query has no rows to give you, so iterating one - or calling eager() on it - throws a \LogicException. Use first(). Note also that asking twice runs the query twice.

Readonly Results

readonly() detaches each entity from the entity manager as it's hydrated. Use it for anything you're only going to read - nothing is tracked for changes, and nothing is flushed:

Aggregates

When your query selects extra columns alongside the entity, withAggregates() wraps each row in an EntityWithAggregates, which proxies to the entity and exposes the extra columns:

[!WARNING] Only call withAggregates() when the query really does select extra columns - iterating throws a \LogicException otherwise. Doctrine can't iterate aggregate results directly, so they're chunked into groups of 20, each requiring an additional query.

Tuning Pagination

Counting and paginating go through Doctrine's Paginator, which is configurable:

Like everything else on an EntityResult, these return a new instance rather than changing the original.

[!NOTE] Output walkers are disabled automatically when a hydration mode or as() modifier is set.

Repositories

ObjectRepository is this package's repository interface, and it is deliberately small: find() for a single object, filter()/query() for an EntityResult, plus count() and iteration.

What's missing is the point. There is no findAll() and no findBy() - nothing in the API hands you an array of entities, so a repository call can't be the thing that exhausts your memory. Anything that returns more than one object returns a lazy EntityResult that you narrow, paginate or batch iterate before it ever touches the database:

[!TIP] find(), filter() and query() all accept specifications - reusable filter objects that are converted into the query itself.

Three implementations are available - they differ only in what your repository also is: nothing else, or a Doctrine repository (in standard and Symfony-autowireable variants).

EntityRepository

The standalone implementation. Use it directly for any entity:

Or extend it for your custom repositories, passing the entity class up to the parent constructor:

[!TIP] The protected qb() helper returns an EntityResultQueryBuilder already scoped to your entity, using the alias you pass it (e by default). It's a Doctrine QueryBuilder, so build the query however you like - ->result() is waiting at the end of the chain.

[!TIP] The protected em() helper gives you the entity manager, for anything the query builder can't do.

[!IMPORTANT] This is not a Doctrine repository. It doesn't extend Doctrine\ORM\EntityRepository and has none of its methods (findAll(), findOneBy(), matching(), ...) - just the ObjectRepository API above. If you want both, use one of the bridges below.

ORMEntityRepository

A Doctrine EntityRepository and an ObjectRepository. Use it when you want the new API without giving up the Doctrine one - findOneBy() and filter() both work:

createQueryBuilder() is overridden to return an EntityResultQueryBuilder, so ->result() is available on the query builders you already build.

[!NOTE] If your repository already extends something else, add the EntityRepositoryBridge trait to it directly - that's all these bridge classes do.

ORMServiceEntityRepository

The same bridge, but extending DoctrineBundle's ServiceEntityRepository so the repository is autowireable in a Symfony application:

Inject it like any other service - see Symfony Integration for autowiring a repository for entities that don't have a repository class at all.

Finding

find() returns a single entity, or null if there isn't one:

Filtering

filter() narrows the repository down to an EntityResult you can iterate, paginate or hydrate:

An EntityResult can be filtered further, but its specifications are converted to a Criteria rather than applied to the query builder, so it understands a smaller set than the repository does:

Querying

query() accepts everything filter() does, and is the one to reach for when the specification changes the database rather than narrowing a read:

Invokable Objects

find(), filter() and query() also accept any invokable object, called with the query builder and the root alias. It's a reusable, testable place to put a query you'd otherwise inline:

[!WARNING] This only applies to repositories. An EntityResult or a bridged collection treats an invokable object as a plain callable(V,K):bool and filters in PHP instead. Wrap it in Spec::callback() if you need it understood by both.

Iterating and Counting

[!WARNING] Iterating a repository directly uses batch iteration: the entity manager is cleared every 100 entities and nothing is ever flushed. Don't hold on to entities from a previous chunk, and don't modify them - those changes are silently discarded.

Batch Processing

Doctrine keeps every entity it hydrates in memory, so a loop over a large table grows until it dies - and mutating one raises the question of when to flush. Batch processing answers both by working in chunks:

[!WARNING] Batch::iterate() never flushes. Changes you make to an entity while iterating are silently discarded when the chunk is cleared - use Batch::process() if you intend to write.

[!IMPORTANT] Batch::process() opens one transaction for the whole loop, not one per chunk. Nothing is left half-done if it fails, but the transaction is held for the entire run - worth keeping in mind when processing very large sets, where it means long-lived locks.

Both take a chunk size, defaulting to 100:

The items can be any iterable, including data that isn't entities at all:

For a Query or QueryBuilder, iteratorFor()/processorFor() wrap it in a Doctrine Paginator first:

[!WARNING] Entities hydrated in earlier chunks are detached once that chunk is done. Don't collect them in an array as you go, and don't hold a reference to one across iterations - re-fetch it instead.

[!TIP] When the batch iterator/processor source is countable, the returned iterator is also countable. This makes it super flexible for use with SymfonyStyle::progressIterate():

[!NOTE] An EntityResult has both built in - $posts->batchIterate() and $posts->batchProcess() do the same thing without needing the entity manager passed in. Both take the same chunk size argument.

Collection Bridge

DoctrineBridgeCollection wraps a doctrine/collections instance and implements both interfaces at once, so it is a Doctrine\Common\Collections\Collection and a Zenstruck\Collection. Everything Doctrine's collection can do still works, and the lazy/paginating/specification API comes along with it:

The interesting part is what happens with an uninitialized relation. Doctrine only needs the whole collection in memory if you make it load - so filtering with a specification becomes a Criteria (executed as a query against the relation), and iterating pages the relation instead of initializing it:

[!NOTE] A Criteria can be passed directly if you prefer it to specifications - both end up in the same place.

[!TIP] This works for in-memory collections too, not just relations: Doctrine's own ArrayCollection is Selectable, so new DoctrineBridgeCollection(['a', 'b']) accepts specifications where ArrayCollection would reject them.

Specifications

A specification is an object that describes a filter or a sort. Unlike a callback, it can be inspected - which is what lets the Doctrine implementations turn it into a query instead of loading everything and filtering in PHP. Build them with the Spec factory:

[!NOTE] Both between() bounds are included unless you say otherwise:

String Wildcards

contains(), startsWith() and endsWith() treat * as a wildcard anywhere within the value:

A leading or trailing * is stripped - the specification already adds one on that side:

* is the only wildcard. SQL's own wildcards are escaped, so they match literally and user input is safe to pass straight through:

[!WARNING] All of this is repository-only. An EntityResult or a bridged collection passes the value straight through to the Criteria, so * matches a literal asterisk and %/_ are left to the database as wildcards.

What Understands What

Specifications go through one of two interpreters, and they don't support the same things:

Anything a source doesn't understand throws Zenstruck\Collection\Exception\InvalidSpecification.

ORM-only Specifications

DoctrineSpec extends Spec, so it's a drop-in replacement that adds specifications only a repository can apply:

Joins can be fetch-joined with eager() and narrowed with scope(), which applies a specification against the joined alias rather than the root one:

Callbacks

Spec::callback() hands you the underlying query object when no specification fits. A repository gives you the query builder and its root alias:

An EntityResult or a bridged collection gives you the Criteria instead. Either modify it directly or return an Expression to have it added for you:

Custom Specifications

Implement Nested to name a specification you keep repeating. Both interpreters unwrap them recursively, so yours works wherever the specifications it's built from work - and composes like any other:

Symfony Integration

Enable the bundle:

There is nothing to configure. When DoctrineBundle is installed, the repository services below are registered automatically.

The bundle also registers the @ZenstruckCollection Twig namespace, which is where the pager templates live:

A Repository For Any Entity

Not every entity deserves its own repository class. ObjectRepositoryFactory builds one on demand for any entity you have:

Or skip the factory entirely and let #[ForObject] inject the repository itself:

[!NOTE] Repositories are cached per entity class, so asking for the same one twice gives you the same instance. The cache is reset between requests in long-running runtimes.

Custom Repositories as Services

EntityRepository - and skip writing a constructor - put #[ForObject] on the class instead:

Inject PostRepository like any other service.

[!WARNING] The attribute works by injecting the entity class into EntityRepository's constructor, so your repository must not define one of its own - the container throws a LogicException when compiling if it does.

[!NOTE] #[ForObject] is an autowiring expression, so it requires symfony/expression-language.

Static Analysis

Everything in this package is generic, and the types survive the whole chain - filtering, hydrating, paginating. This library is analyzed at PHPStan level 8 and ships the annotations for your code to be as well.

Collection<V,K> is templated on both its values and its keys, but the key defaults to array-key, so Collection<Post> is usually all you need to write:

collect() narrows to the implementation it actually returns, so you keep the extra API of whatever you passed it:

The Doctrine types follow the same rule - what you hydrate is what you get back:

Recipes

Three passes at the same problem - iterating GitHub's issue search, which hands out 100 results at a time and reports a total_count - each one fixing something the previous one couldn't do cheaply.

Iterating a Paginated API

Wrapping the "keep requesting until there are none left" loop in a LazyCollection turns it into something you can iterate, filter and map, and nothing is requested until you do:

Pages are requested as you iterate, and stop being requested when you stop:

Two things are expensive, though:

count() has to walk every page to arrive at a number, and take() reaches an offset by skipping items from the front - so paginate() gets slower the deeper you go: page 1 costs one request, page 11 costs three, page 50 costs eleven. The next two recipes deal with each in turn.

Counting It Without Walking It

The API reports the total itself, so hand it over with CallbackCollection - the second callback is only ever called by count():

That's the first problem gone. The second one remains: take() still skips from the front, so $issues->take(20, 980) is still eleven requests.

Wrapping It in Your Own Collection

Implementing Collection yourself is mostly free - the IterableCollection trait supplies everything except getIterator() - and it lets you override take() as well, which is the one thing the previous two recipes can't do:

Both of the expensive operations are now a single request, however deep you reach:

[!TIP] paginate() is built on take(), so it inherits this: any page costs one request no matter how deep, and a Full pager gets its total from count() for one more.

[!NOTE] @implements Collection<array<string,mixed>,int> is what keeps this type-safe - PHPStan knows what first() returns and what the callbacks receive. Careful with helper names too: pages() is already part of the interface, which is why the private one above is apiPages().

Security Policy

If you discover a security vulnerability, please do not open a public issue or pull request. Instead, please review this repository's Security Policy for instructions on how to report it responsibly.