Download the PHP package nerou/large-array-buffer without Composer
On this page you can find all versions of the php package nerou/large-array-buffer. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
FAQ
Example:
In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
- Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
- To use Composer is sometimes complicated. Especially for beginners.
- Composer needs much resources. Sometimes they are not available on a simple webspace.
- If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
- Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Informations about the package large-array-buffer
LargeArrayBuffer
This is for you, if...
...you are working with pretty large arrays that conflict with close to every memory limit you can set. The array elements on the other hand should not be that big, such that you can keep a single element in memory quite easily. If the array gets too big for memory, it will be moved to disk temporarily and transparently. You can still iterate over it as if it was a normal array.
One typical use case would be to load a lot of datasets from a database at once. (There are reasons to prefer this over running multiple queries.) See Usage below for an example for this use case using this library.
Install
Note: This library requires PHP 8.0+!
Use composer to install this library:
composer require nerou/large-array-buffer
There are pretty much no dependencies with some exceptions:
- If you want to use the
toJSONFile()method, you need to installext-json(PHP's PECL JSON extension) as well. - If you want to use the igbinary serializer,
ext-igbinaryis required. See php-ext-igbinary. - If you want to use the msgpack serializer,
ext-msgpackis required. See php-ext-msgpack. - If you want to use LZ4 compression,
ext-lz4is required. See php-ext-lz4.
Usage
Options
The constructor of LargeArrayBuffer provides some options:
- You can set the threshold when to move the data to disk. When pushing data to the buffer, it is stored in memory until it gets too large.
E.g.:
new LargeArrayBuffer(512);to set a 512 MiB threshold. - You can choose either the PHP serializer, the igbinary serializer or the msgpack serializer (PHP serializer is default).
E.g.:
new LargeArrayBuffer(serializer: LargeArrayBuffer::COMPRESSION_IGBINARY); - You can enable GZIP or LZ4 compression for the serialized items. Although this is recommended only if your items are pretty big like > 1 KiB each. E.g.:
new LargeArrayBuffer(compression: LargeArrayBuffer::COMPRESSION_GZIP);. Note, that LZ4 compression requires ext-lz4 to be loaded.
Read from the buffer
There are several options to read the data:
- Iterate: As you might have seen in the example above, you can iterate over the buffer just like over an array with
foreach.foreach($buffer as $item){ /* work with your data here */ } toArray(): If you have a set of buffers which do not fit in memory all together but one at a time, you can use$buffer->toArray()to get an array to work with.toJSONFile(): If you want to write the array in JSON format to some file orresource, use this method. It supports all thejson_encode()options and flags.
Buffer stats
There are some stats you can obtain:
count(): The number of items in the buffer.getSize(): The number of bytes of the serialized (!) data.
How it works
To put it in one sentence: This library uses php://temp as well as PHP's serialize/unserialize functions to store an array on disk if it gets too large.
Limitations and concerns
- associative arrays are not supported
- the item type needs to be compatible with PHP's serialize/unserialize functions
- since storage drives (even PCIe SSDs) are a lot slower than memory and de-/serialization needs to be done, you trade hard memory overflows for performance losses
Benchmark
A benchmark with 1 million measurements (consisting of DateTimeImmutable, int and float) using PHP 8.1 with 10 iterations comparing a normal array with the LargeArrayBuffer gave the following results (LargeArrayBuffer was configured with a memory limit of 128 MiB):
| Action | Serializer | Compression | Consumed time | Consumed memory | Buffer size |
|---|---|---|---|---|---|
| Fill array | none | none | 1.57 s | 490.0 MiB | NA |
| Iterate over array | none | none | 0.27 s | 492.0 MiB | NA |
| Fill buffer | PHP | none | 4.27 s | 0 B | 378.7 MiB |
| Iterate over buffer | PHP | none | 2.85 s | 0 B | 378.7 MiB |
| Fill buffer | PHP | GZIP | 24.76 s | 0 B | 192.5 MiB |
| Iterate over buffer | PHP | GZIP | 6.79 s | 0 B | 192.5 MiB |
| Fill buffer | PHP | LZ4 | 4.89 s | 0 B | 241.0 MiB |
| Iterate over buffer | PHP | LZ4 | 2.93 s | 0 B | 241.0 MiB |
| Fill buffer | igbinary | none | 4.26 s | 0 B | 319.1 MiB |
| Iterate over buffer | igbinary | none | 3.41 s | 0 B | 319.1 MiB |
| Fill buffer | igbinary | GZIP | 21.50 s | 0 B | 173.2 MiB |
| Iterate over buffer | igbinary | GZIP | 4.80 s | 0 B | 173.2 MiB |
| Fill buffer | igbinary | LZ4 | 4.38 s | 0 B | 195.1 MiB |
| Iterate over buffer | igbinary | LZ4 | 3.17 s | 0 B | 195.1 MiB |
Note:
- The peak memory usage using the buffer is about its memory limit. The table shows the memory usage after the specified action.
- PHP seems to cache the array once it is created for the first time, although
unsetis used. That is why I have not put the average value in the table for this specific value but the maximum (first run). - The serialized data is smaller than the binary data in memory. I have absolutly no idea why.
To reproduce call bench/benchmark.php.
License
This library is licensed under the MIT License (MIT). Please see LICENSE for more information.
