Download the PHP package survos/jsonl-bundle without Composer

Survos JsonlBundle

Streaming JSONL read/write utilities for Symfony, with first-class support for resumable writes, sidecar progress tracking, transparent gzip compression, and CLI-level inspection and querying.

JsonlBundle is intentionally small at the API surface, but feature-rich under the hood. Many of its most powerful capabilities are currently “hidden” because they require no extra code once you adopt the bundle’s reader/writer abstractions.

This README makes those capabilities explicit, with concrete, copy-pasteable examples.

Why JSONL?

JSON Lines (.jsonl) is the ideal format for:

• Large datasets
• Streaming ingestion pipelines
• Fault-tolerant ETL
• CLI-driven workflows
• Append-only logs
• Partial resume after failure

JsonlBundle embraces these properties rather than fighting them.

Installation

Symfony Flex will register the bundle automatically.

Core Concepts

Files

• One JSON object per line
• Append-only
• Safe for streaming

Sidecar

Every JSONL file written with JsonlWriter has an optional sidecar file that tracks:

• Total records written
• Byte offset
• Completion status
• Timestamps
• Resume metadata

Sidecars enable resume, progress inspection, and CLI tooling.

Compression

If the filename ends in .gz, compression is automatic.

No flags. No config. Just naming.

Writing JSONL

Basic Write (Overwrite by Default)

By default, JsonlWriter::open() uses overwrite mode ('w'), matching PHP’s fopen() semantics.

This means:

• Existing files are truncated
• Sidecar state is reset
• You are explicitly starting a new dataset

This immediately gives you:

• File locking
• Atomic writes
• Sidecar tracking

Append vs Overwrite

JsonlBundle makes dataset intent explicit via write modes.

• 'w' — overwrite (default): start a fresh dataset
• 'a' — append: continue an existing dataset

Append to an Existing Dataset

Use append mode when resuming or extending a JSONL file:

Append mode:

• Preserves existing data
• Resumes safely using the sidecar
• Is safe to re-run after partial failure

Overwrite an Existing Dataset

Overwrite mode is intentional and destructive:

Overwrite mode:

• Truncates the file
• Resets sidecar state
• Signals a brand-new dataset

No buffering required. Memory-safe for large datasets.

Resume a Partial Write

If the process crashes midway, reopen the file in append mode:

The writer will:

• Detect the sidecar
• Resume safely
• Avoid corrupt output

Writing with Gzip Compression

Just use .gz — mode semantics are unchanged:

Features retained:

• Streaming
• Resume
• Sidecar
• Locking

Writer Options

Advanced behavior is controlled via JsonlWriterOptions, passed as the third argument to JsonlWriter::open().

Options configure policy, not mechanics. Defaults are safe and explicit.

Example: Overwrite and Ensure Directory Exists

In this example:

• 'w' signals a fresh dataset
• ensureDir: true allows directory creation explicitly

Avoid overwrite mode ('w') for resumable pipelines. Append mode with sidecars is the intended default for long-running jobs.

For the full list of writer options and their semantics, see:

• doc/basic.md
• doc/advanced.md

Reading JSONL

Basic Read

The reader:

• Streams line-by-line
• Uses constant memory
• Works for .jsonl and .jsonl.gz

Read with Offset (Resume Processing)

Ideal for batch pipelines and workers.

Sidecar Files (The Hidden Superpower)

For a file:

JsonlBundle maintains:

What’s Inside the Sidecar?

Typical contents:

You do not manage this manually.

Query Progress Programmatically

Mark Completion Explicitly

This is especially useful in multi-stage pipelines.

CLI Utilities

JsonlBundle ships with CLI tooling to inspect files without writing code.

Inspect a File

Example output:

Inspect a Gzipped File

Works exactly the same.

Peek at Records

Resume-Aware Counting

Unlike wc -l, JsonlBundle uses the sidecar:

Fast and accurate even for huge files.

Patterns Enabled by JsonlBundle

ETL Pipelines

Each step emits JSONL, resumes safely, and records progress.

Doctrine Import Pipelines

• Stream JSONL
• Batch insert entities
• Resume on failure
• Track progress via sidecar

Translation Caches

• source.jsonl
• target.jsonl
• De-duplication
• Resume after API failure

Search Index Feeds

• Emit JSONL once
• Replay into Meilisearch / OpenSearch
• Deterministic re-runs

When Not to Use JsonlBundle

• Small datasets fully loaded into memory
• Interactive CRUD forms
• Request/response APIs

JsonlBundle shines in pipelines, CLI tools, and long-running jobs.

Design Philosophy

• Zero configuration
• Filename-driven behavior
• Streaming first
• Fail-safe defaults
• Sidecar over database state
• CLI-friendly

Summary

JsonlBundle is more than a reader/writer:

• ✔ Resumable writes
• ✔ Sidecar progress tracking
• ✔ Transparent gzip compression
• ✔ CLI inspection & querying
• ✔ Streaming everywhere

Most of this works automatically once you adopt JsonlWriter and JsonlReader.

Example: Caching a Remote API as JSONL (DummyJSON Products)

This example demonstrates how JsonlBundle can be used as a simple, durable cache for a remote HTTP API, without introducing a separate caching abstraction.

Goal:

• Command: bin/console products:list
• If products.jsonl does not exist:
  • Fetch https://dummyjson.com/products
  • Write each product as a JSONL record
• Then:
  • Read products.jsonl
  • Display the first product

This turns a one-shot HTTP call into a replayable, CLI-friendly dataset.

The Command (Minimal, Accurate API Usage)

What This Example Actually Uses

This command relies only on existing, stable APIs:

JsonlWriter

• JsonlWriter::open($filename)
• ->write(array $row)
• ->close()

JsonlReader

• JsonlReader::open($filename)
• foreach ($reader as $row)

No hidden methods. No imaginary helpers.

What You Still Get (Implicitly)

Even with this minimal API surface, you still benefit from:

• ✔ Streaming writes (no memory pressure)
• ✔ File locking
• ✔ Safe append semantics
• ✔ Sidecar tracking (rows / bytes / timestamps)
• ✔ Resume-safe re-execution
• ✔ Transparent .gz support via filename

None of this appears in the command — which is the point.

Optional Variations

Enable Compression (Zero Code Changes)

Inspect the Cache from the CLI

Re-run Offline

Once written, the command works without network access.

Why This Pattern Matters

This example shows the intended JsonlBundle workflow:

• Treat JSONL as a first-class artifact
• Use filenames, not flags, to control behavior
• Let sidecars track progress instead of databases
• Make pipelines restartable by default

You are not “caching HTTP”. You are materializing data.

That distinction is why JsonlBundle scales cleanly from demos to production pipelines.

Inspecting JSONL Files (jsonl:info)

In addition to counting rows, JsonlBundle provides a command to inspect progress and status metadata stored in sidecar files.

Show info for a single file

Example output:

This is the fastest way to answer questions like:

• “How many records are written so far?”
• “Did this pipeline finish?”
• “When was this dataset last updated?”

Inspect a directory of JSONL files

Recurse into subdirectories

Directory mode renders a table with one row per file, including completion status and timestamps:

Why jsonl:info matters

• Uses sidecar metadata, not slow file scans
• Works for .jsonl and .jsonl.gz
• Distinguishes partial vs completed datasets
• Ideal for long-running or resumable pipelines

For deeper discussion of sidecars, resume logic, and CLI workflows, see the advanced usage guide.

Advanced Usage

This bundle is intentionally small at the surface but powerful in real-world pipelines.

For production patterns—including resume semantics, sidecar files, row counting, CLI workflows, and anti-patterns to avoid—see the advanced documentation:

👉 More examples 👉 Advanced Usage Guide

That document is where long-running jobs, API dumps, and restartable ingestion pipelines are covered in detail.

Gzip compression

The filename chooses the format: .jsonl is plain text; .jsonl.gz is gzip. Gzip defaults to level 1 (fast), including outside Symfony. Level 0 still writes a gzip stream; it does not mean a plain file.

JsonlWriter::open($path, compressionLevel: 6) overrides the configured default for that writer. Existing open writers keep their original level.

For applications with an optional-compression policy, use Jsonl::outputPath($path, compression: false) for plain JSONL, or pass an integer 0–9 for a .gz suffix and the same integer to open(compressionLevel: ...). Jsonl::resolvePath() and JsonlReader::open() find a gzip sibling when the plain path is absent; Jsonl::files() discovers both formats without duplicate cores. When both exist, the plain file wins so a new normalization cannot reuse an older compressed copy.

Reading JSONL inside a local archive

The reader can stream plain or gzipped JSONL members from a local TAR (including an uncompressed BagIt TAR), without extracting its payload. Concatenated gzip members are supported; byte offsets refer to the combined uncompressed stream. BagIt manifests and archive integrity must be verified by the application before using the input. This reader does not download S3 objects or validate BagIt.

Existing phar:///absolute/raw.bagit.tar/data/raw/obj.jsonl.gz paths also work with JsonlReader::open(). Ordinary file reads retain their existing behavior.

State and indexes remain writable local SQLite sidecars. Pass an existing cache directory explicitly; the library does not create application directories. The sidecar filename combines the archive SHA-256 and member-path hash, so distinct members and archive versions cannot accidentally reuse each other's indexes. Reopen the reader after replacing an archive; archive inputs are immutable during an open reader's lifetime.

The optional persisted row cache makes lookups independent of gzip seeking. If that cache is removed, the index still locates the uncompressed offset, but gzip requires a sequential scan to that offset. Sequential iteration always reads the archive payload, not the sidecar. Archive state/index access without an explicit local cache directory fails clearly rather than trying to write into the TAR.

This adds archive support to the reader, line counters, and index builder. Other commands that independently open files or manage sidecars need to opt into these APIs; archive support is not a claim that every command already accepts archives.