Download the PHP package entreya/csvquery without Composer
On this page you can find all versions of the php package entreya/csvquery. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Download entreya/csvquery
More information about entreya/csvquery
Files in entreya/csvquery
Package csvquery
Short Description High-performance CSV query engine with Go-powered indexing and Yii2-like fluent API
License MIT
Homepage https://github.com/entreya/csvquery
Informations about the package csvquery
CsvQuery
Query massive CSV files like a database β powered by a Go sidecar engine.
CsvQuery is a high-performance query engine that turns CSV files (10 GBβ1 TB+) into searchable, indexed data stores. A fluent PHP API (inspired by Yii2 ActiveQuery) communicates with a Go sidecar daemon over Unix Domain Sockets, delivering sub-millisecond query latency without the overhead of a traditional database.
π Full documentation β entreya.github.io/csvquery-docs
Table of Contents
- Key Features
- Architecture
- Installation
- Quick Start
- Usage Examples
- API Reference
- Data Flow
- Performance
- CLI Reference
- Project Structure
- Testing
- Contributing
- License & Acknowledgments
β¨ Key Features
| Feature | Description |
|---|---|
| π SIMD-Accelerated Parsing | AVX2 / SSE4.2 instructions scan CSV delimiters at hardware speed (10 GB/s+) |
| π Massive Scale | Benchmarked on 18 M+ rows, 10 GB+ files |
| πΎ Memory Efficient | mmap-based file access with LZ4-compressed indexes |
| π Yii2-like Fluent API | Familiar find()βwhere()βall() query builder for PHP developers |
| β‘ Zero-IO Index Scans | COUNT(*) operations resolve from index metadata alone β the CSV is never opened |
| π Live Updates | Sidecar _updates.json system makes immutable CSVs mutable |
| π Daemon Mode | Persistent Go process over UDS for ~1 ms latency (vs ~200 ms per-process spawn) |
| π Cross-Platform | Pre-compiled binaries for Linux, macOS, and Windows (AMD64 & ARM64) |
ποΈ Architecture
CsvQuery uses a PHP + Go Sidecar design. Your PHP application talks to a long-lived Go daemon through Unix Domain Sockets (UDS), keeping query latency under a millisecond.
Key Technologies
| Component | Technology | Why |
|---|---|---|
| Parsing | AVX2 / SSE4.2 SIMD | Scan delimiters at hardware speed |
| Compression | LZ4 block codec | 10Γ faster decompression than Gzip |
| File Access | mmap |
Zero-copy reads, OS-managed page cache |
| IPC | Unix Domain Sockets | ~1 ms round-trip vs ~200 ms process spawn |
| Probabilistic Filter | Bloom filters | Reject non-matching index blocks before decompression |
Design Patterns
| Pattern | Where | Purpose |
|---|---|---|
| Sidecar Process | Go daemon | Offload CPU-intensive work from PHP |
| Fluent Builder | ActiveQuery |
Chainable, composable query construction |
| Singleton | SocketClient |
Reuse a single daemon connection per request |
| Generator / Streaming | each(), GoBridge::query() |
Process millions of rows without loading them all into memory |
| External Merge Sort | Indexer | Build indexes on files larger than available RAM |
π¦ Installation
Prerequisites
| Requirement | Version |
|---|---|
| PHP | 8.1+ |
| Composer | 2.x |
| Go (optional β pre-compiled binaries included) | 1.21+ |
Via Composer
The Go binary is compiled automatically on composer install. If Go is not available on your machine, the bundled pre-compiled binary for your platform will be used instead.
Build Commands
Manual Build
Platform Notes
| Platform | Notes |
|---|---|
| macOS | ARM64 (Apple Silicon) and AMD64 binaries included |
| Linux | AMD64 and ARM64 binaries included |
| Windows | Statically linked .exe β no MinGW / Cygwin / WSL required |
See CONTRIBUTING.md for per-platform development setup.
π Quick Start
π Usage Examples
Filtering with Operators
Complex Nested Conditions
Aggregations & Group By
Data Mutation
Query Debugging
Efficient Iteration
π API Reference
CsvQuery β Entry Point
Constructor Options
| Option | Type | Default | Description |
|---|---|---|---|
indexDir |
string |
CSV directory | Directory to store .cidx / .bloom files |
separator |
string |
, |
CSV column delimiter |
workers |
int |
CPU count | Parallel workers for indexing |
memoryMB |
int |
500 |
Memory limit per worker (MB) |
binaryPath |
string |
Auto-detect | Custom path to the Go binary |
Methods
| Method | Returns | Description |
|---|---|---|
find() |
ActiveQuery |
Start a new query |
where($condition, $value) |
ActiveQuery |
Shorthand for find()->where(...) |
andWhere($condition, $value) |
ActiveQuery |
Shorthand for find()->andWhere(...) |
createIndex(array $columns, bool $verbose, array $options) |
bool |
Create indexes (single or composite) |
hasIndex(string\|array $column) |
bool |
Check whether an index exists |
dropIndex(string\|array $column) |
bool |
Remove a specific index |
clearIndexes() |
int |
Remove all indexes; returns count deleted |
getHeaders() |
array |
Column names from the CSV header row |
insert(array $row) |
void |
Append a single row |
batchInsert(array $rows) |
void |
Append multiple rows |
update(array $attributes, array $conditions) |
int |
Update matching rows via sidecar; returns count |
addColumn(string $name, string $default) |
void |
Add a virtual column |
getMeta() |
array |
Index metadata (row count, file hash, etc.) |
validateIntegrity() |
bool |
Check if indexes are still valid for the current CSV |
ActiveQuery β Query Builder
Condition Methods (return $this for chaining)
| Method | Description |
|---|---|
where($condition, $value = null) |
Set the WHERE clause |
andWhere($condition, $value = null) |
Append an AND condition |
orWhere($condition, $value = null) |
Append an OR condition |
filterWhere(array $condition) |
WHERE, ignoring empty/null values |
andFilterWhere(array $condition) |
AND, ignoring empty/null values |
orFilterWhere(array $condition) |
OR, ignoring empty/null values |
Modifier Methods (return $this for chaining)
| Method | Description |
|---|---|
select(array $columns) |
Choose which columns to return |
orderBy(array\|string $columns) |
Set ORDER BY |
addOrderBy(array\|string $columns) |
Append additional ORDER BY |
groupBy(array\|string $columns) |
Set GROUP BY |
addGroupBy(array\|string $columns) |
Append additional GROUP BY |
limit(int $n) |
Maximum rows to return |
offset(int $n) |
Skip first n rows |
indexBy(string\|callable $column) |
Key results by a column value |
asArray(bool $flag = true) |
Return plain arrays instead of Row objects |
debug(bool $enable = true) |
Enable debug mode on the bridge |
Execution Methods
| Method | Returns | Description |
|---|---|---|
all() |
array |
All matching rows |
one() |
?Row |
First matching row |
count(string $q = '*') |
int |
Count of matching rows |
sum(string $column) |
float |
Sum of column values |
average(string $column) |
float |
Average of column values |
min(string $column) |
mixed |
Minimum column value |
max(string $column) |
mixed |
Maximum column value |
each() |
Generator |
Stream rows one at a time |
aggregate(string $col, array $fns) |
array |
Multiple aggregations in one pass |
explain() |
array |
Query execution plan |
createCommand() |
Command |
SQL-like debug representation |
Condition Syntax
Row β Result Object
Implements ArrayAccess, IteratorAggregate, and JsonSerializable.
Cell β Value Wrapper
Column β Column View
π Data Flow
Query Lifecycle
Index Creation
Query Strategy Decision
β‘ Performance
Benchmarks on 1,000,000 rows (Darwin ARM64, Apple M-series):
Indexing Throughput
| Task | Throughput | Wall Time |
|---|---|---|
| Single-column index | ~400,000 rows/sec | ~2.5 s |
| Composite index (2 cols) | ~350,000 rows/sec | ~2.9 s |
Query Latency
| Query Type | Matching Rows | Latency |
|---|---|---|
COUNT(*) β no filter |
all | ~10 ms |
COUNT β indexed filter (0 hits) |
0 | ~14 ms |
COUNT β indexed filter |
150 K | ~25 ms |
SELECT β indexed filter |
1 K | ~50 ms |
| Full table scan (no index) | all | ~2,000 ms |
[!TIP] Zero-IO Index Scans β If the query can be satisfied entirely by index metadata (e.g.
COUNT(*)), the engine never opens the CSV file.
π οΈ CLI Reference
The Go binary can be invoked directly for maintenance, debugging, or scripting:
Commands
index β Create indexes from a CSV file
| Flag | Default | Description |
|------|---------|-------------|
| `--input` | *(required)* | Path to CSV file |
| `--output` | CSV directory | Output directory for index files |
| `--columns` | `[]` | JSON array of columns to index |
| `--separator` | `,` | CSV delimiter |
| `--workers` | CPU count | Parallel workers |
| `--memory` | `500` | Memory limit per worker (MB) |
| `--bloom` | `0.01` | Bloom filter false-positive rate |
| `--verbose` | `false` | Print progress |
query β Execute queries
| Flag | Default | Description |
|------|---------|-------------|
| `--csv` | | Path to CSV file |
| `--index-dir` | CSV directory | Index directory |
| `--where` | `{}` | JSON conditions |
| `--limit` | `0` (unlimited) | Max results |
| `--offset` | `0` | Skip first *n* results |
| `--count` | `false` | Output only the count |
| `--explain` | `false` | Print query execution plan |
| `--group-by` | | Column to group by |
| `--agg-col` | | Column to aggregate |
| `--agg-func` | | Aggregation function |
daemon β Start the UDS server
| Flag | Default | Description |
|------|---------|-------------|
| `--socket` | `/tmp/csvquery.sock` | Unix socket path |
| `--host` | `127.0.0.1` | TCP host (if `--port` is set) |
| `--port` | `0` | TCP port (0 = use Unix socket) |
| `--csv` | | Default CSV path |
| `--index-dir` | | Default index directory |
| `--workers` | `50` | Max concurrent handlers |
write β Append rows to a CSV file
| Flag | Default | Description |
|------|---------|-------------|
| `--csv` | *(required)* | Target CSV file |
| `--headers` | `[]` | JSON array of headers (new file only) |
| `--data` | `[]` | JSON array of row arrays |
| `--separator` | `,` | CSV delimiter |
version β Print version
π Project Structure
See ARCHITECTURE.md for detailed module documentation.
π§ͺ Testing
PHP
Go
π€ Contributing
Contributions are welcome! Please see CONTRIBUTING.md for guidelines.
π License & Acknowledgments
This project is licensed under the MIT License β see the LICENSE file for details.
Acknowledgments:
- Query API inspired by Yii2 ActiveQuery
- LZ4 compression via pierrec/lz4
- SIMD optimizations inspired by simdjson
All versions of csvquery with dependencies
ext-json Version *