Download the PHP package mikibuilder/llm-vcr without Composer

llm-vcr

English · Español

Semantic record & replay for AI features in PHP.

Record what your LLM actually returns, replay it in CI with no network and no API key, and find out when your provider silently changes the model and breaks your DTOs.

The problem

You have a service that classifies support tickets with an LLM. You want to test it. And then:

1. It is not deterministic. The same prompt returns different text every time.
2. It costs money. 200 tests × every push × every developer.
3. It is slow. Between 0.5 and 3 seconds per call. Your suite goes from seconds to minutes.
4. It needs network access and a production API key in CI. If your provider has an incident, your build turns red without you breaking anything.
5. And the worst one: silent drift. The provider updates the model, urgency starts coming back as "high" instead of 4, your typed DTO blows up in production — and you never touched a single line of code. No test catches it, because your mocks have the old value frozen in.

The solution

That is it. Your code does not change: RecordingPlatform implements the same interface.

• Locally it records real responses into versionable JSON files.
• In CI it replays them from disk: no network, no API key, no cost.
• Every night it replays them against the real provider and warns you if anything changed.

What makes it different

The key insight: php-vcr matches requests by exact hash. A real prompt carries timestamps, UUIDs and IDs that change on every run, so the cassette is invalidated the moment you touch a comma. llm-vcr normalises that noise and compares using cosine similarity.

Installation

Requires PHP 8.2+ with ext-json and ext-mbstring. No runtime dependencies.

Get started in 2 minutes (no sign-up required)

The demo shows all six behaviours using a simulated platform. No API key, no network.

With a real, free LLM

Groq gives you a free API key with no credit card (30 req/min, ~1,000 per day on the free tier).

With Docker

Symfony integration

The bundle adds declarative configuration and a Web Profiler panel with per-request metrics.

Symfony is an optional dependency: if you only use PHPUnit or Pest, you pull in nothing.

And in your service:

The Profiler panel

The debug toolbar shows at a glance how many invocations came from disk and how many hit the API. The panel breaks it down:

The badge turns red if live calls were made while in replay mode: usually it means a cassette is missing.

Console command

It exits with code 1 if it detects HIGH or CRITICAL drift, so you can chain it into a nightly cron and break the build.

It needs your LLM client registered under the llm_vcr.live_platform alias:

PHPUnit and Pest integration

The goal is that setting up an LLM test takes one line, and that assertions speak the language of the problem instead of forcing you to write plumbing.

PHPUnit — the InteractsWithLlm trait

No paths to configure: cassettes go to <test-directory>/cassettes/ and the name is derived from the class and method (ticket--classifies-an-access-problem.json).

Pest — native expectations

They register themselves when you install the package: no need to touch Pest.php. Available: toBeLlmJson(), toMatchLlmShape(), toHaveLlmValueIn(), toComeFromCassette(), toHaveMadeNoLiveCalls(), toHaveReplayed().

Why assert on shape instead of value: the value an LLM returns is not deterministic, but the contract must be. toMatchLlmShape() fails when urgency goes from int to string — which is exactly the bug that breaks your DTO in production.

Prompts with dynamic parameters

Three strategies, from strictest to most tolerant:

PlaceholderMatcher is the middle ground most people actually want: still an exact, deterministic comparison, reviewable in a PR, but immune to the data you mark as variable. If something you did not declare changes, it does not match — and that is the correct behaviour. Dates, times and UUIDs are covered out of the box.

Usage

The four modes

Run once with LLM_VCR_MODE=record, commit the cassettes, and from then on CI runs for free.

Drift detection

Real output:

That int -> string is the bug that would have cost you a 3 a.m. page.

The .github/workflows/drift.yml workflow runs it nightly and opens an issue automatically.

Configuration

Redaction

Enabled by default, because cassettes get committed to git.

Detects: OpenAI/Groq/GitHub/AWS keys, JWTs, Bearer tokens, emails, phone numbers, national ID numbers, IBANs and card numbers.

Other providers

GroqPlatform speaks the OpenAI dialect, so it works with any compatible endpoint:

For anything else, implement PlatformInterface — it is three lines.

How it works

It is a decorator, not a fork. Pure Liskov substitution: it wraps any implementation of PlatformInterface without your business code noticing.

A cassette from the inside

Readable, diffable in a PR, and without a single secret.

FAQ

Should I commit the cassettes? Yes. They are the project's fixtures: without them, CI cannot run in replay mode. That is exactly why redaction is on by default.

What if I change the prompt? If the change is small, the semantic matcher absorbs it. If it is large, the test fails with a message telling you exactly what to do. Re-record with LLM_VCR_MODE=record and commit.

Does it replace evals? No, they are complementary. Evals measure quality (is the answer good?). llm-vcr solves determinism, cost and drift. You can use both.

Placeholders or semantic similarity? Start with PlaceholderMatcher if you know exactly which parts of the prompt vary (IDs, amounts, business dates): it is deterministic and produces no false positives. Use SemanticMatcher when the prompt is worded differently each time or generated by another system.

Does it work with Symfony AI? Yes. RecordingPlatform is a decorator over a minimal interface, so a three-line adapter is enough. A native bundle is included for the rest of the integration.

Why not use embeddings for matching? Because it would require a network call on the very path that is trying to avoid one. Cosine similarity over normalised bag-of-words works surprisingly well and is instant. An optional EmbeddingMatcher is on the roadmap.

Roadmap

• [x] Core: record/replay, semantic matching, redaction, drift detection
• [x] llm-vcr drift CLI with Markdown output for PRs
• [x] GitHub Actions: CI in replay mode + nightly drift cron
• [x] InteractsWithLlm trait for PHPUnit with 6 assertions
• [x] Native Pest expectations (verified against Pest 3)
• [x] PlaceholderMatcher for prompts with dynamic parameters
• [x] LlmVcrBundle for Symfony, with Web Profiler panel
• [ ] EmbeddingMatcher with on-disk cache
• [ ] Support for streaming responses and tool calls

Contributing

PRs are welcome. The bar: PHPStan level 9 and green tests.

License

MIT — MikiBuilder