Download the PHP package redberry/pest-plugin-evals without Composer

Pest Plugin Evals

A Pest plugin for evaluating LLM outputs. Test your Laravel AI agents with expressive, readable assertions.

• Effortless — Write evals like regular Pest tests. No boilerplate, no ceremony.
• Laravel AI Native — Direct integration with Laravel AI SDK Agent classes.
• Magic Judges — Pass a plain-English string and get LLM-based evaluation. No setup required.

What Is This?

Pest Plugin Evals lets you evaluate the quality of your AI agent outputs inside your Pest test suite. You write a prompt, run the agent, and assert that the output is good — using plain English criteria, deterministic checks, or both.

It works with Laravel AI SDK agents. If you've built an agent class that extends Laravel's Agent contract, this plugin can test it. You don't need to know anything about the AI SDK internals — just pass your agent class and a prompt.

Requirements: PHP 8.3+, Laravel with the AI SDK, Pest 4.

Table of Contents

• Installation
• Quick Start
• The evaluate() Function
• Sending Prompts
  • Simple Prompt
  • Provider, Model & Timeout Overrides
  • Attachments
  • Using EvalCase
• Assertions
  • LLM-as-a-Judge Assertions
  • Deterministic Assertions
  • Tool Assertions
  • Structured Output Assertions
• Sampling
• Datasets
  • Inline EvalCase
  • JSON Datasets
  • XML Datasets
  • Directory Auto-Discovery
• Custom Judges
  • Rubric Classes
  • Judge Classes
  • Custom Judge Instructions
• Configuration
• CLI Output
• Running in CI/CD
• Full Examples
• API Reference

This package is built and maintained by Redberry, one of the few Official Premier Laravel Partner agencies worldwide. With 250+ Laravel projects shipped across 20+ countries, a 200-person team, and over a decade in the Laravel ecosystem, Redberry has helped startups, SMEs, and publicly traded enterprises in regulated industries build SaaS platforms, custom web applications, APIs, and more. Learn about our AI development services.

Installation

The plugin auto-registers with Pest and Laravel via the service provider.

If you want to customize the default judge, output, or sampling settings, publish the config file:

This will create config/evals.php in your application.

Quick Start

Create a test file (e.g. tests/Evals/PostWriterTest.php) and write your first eval:

That's a full, working eval. Here's what happens when you run it:

1. The plugin resolves your PostWriter agent from Laravel's service container.
2. It sends the prompt "Write a blog post about Laravel" to the agent.
3. An LLM judge reads the agent's response and decides if it meets the criterion "The content is engaging and informative".
4. Pest reports pass or fail.

You can mix LLM-based checks with classic deterministic ones in the same test:

A note on syntax: This guide uses BDD-style methods (whenPrompted, toMeet, toBeSimilarTo, etc.) as the primary syntax. Every BDD method has a traditional equivalent (prompt, assertMeets, assertSimilarTo, etc.). We'll point these out as we go.

The evaluate() Function

Every eval starts with evaluate(). It accepts your agent in several forms:

All four forms produce the same thing: an EvalBuilder that you chain prompts and assertions onto.

Sending Prompts

Simple Prompt

Use whenPrompted() to send a prompt to your agent:

whenPrompted() is a BDD alias for prompt(). They are interchangeable.

Provider, Model & Timeout Overrides

You can override the agent's default provider, model, or timeout. Pass them as named parameters to prompt():

Or use separate fluent methods. These set defaults that prompt() parameters can override:

Attachments

For agents that process files or images, pass attachments inline with prompt() or via a separate method:

Using EvalCase

An EvalCase bundles a prompt, expected output, and attachments into one reusable object. Load it with withCase():

Only prompt is required. expected and attachments are optional. You'll see much more about EvalCase in the Datasets section.

Assertions

Assertions check the agent's output. There are four kinds:

1. LLM-as-a-Judge — An LLM reads the output and judges its quality. Powerful but costs an API call.
2. Deterministic — Classic checks like "contains this string" or "shorter than 280 characters". Fast, free, no AI involved.
3. Tool — Checks which tools (e.g., web search, database lookup) the agent called and with what arguments.
4. Structured Output — Checks keys, values, and shape of array/object output from agents that return structured data.

You can mix all four kinds in a single test chain.

LLM-as-a-Judge Assertions

This is the plugin's most powerful feature. You describe what "good" looks like in plain English, and an LLM (the "judge") decides if the agent's output meets that bar.

Pass/Fail Check

toMeet() sends your criterion to the judge and expects a pass:

toMeet() is a BDD alias for assertMeets(). They are interchangeable.

Scored Check

Pass a threshold (0-100) as the second argument. The judge scores the output, and it must meet or exceed the threshold:

Negation

Check that output does not meet a criterion:

There is no BDD alias for assertDoesNotMeet() — use it directly.

Similarity Check

Compare the agent's output against an expected value for semantic similarity. The judge scores how similar they are:

You can customize the similarity threshold (default is 80):

toBeSimilarTo() is a BDD alias for assertSimilarTo().

If you set the expected value separately with ->expected(), use toBeSimilar() (no argument):

toBeSimilar() is a BDD alias for assertSimilar().

Exact Match

toBe() does a deterministic exact comparison. It auto-detects the type:

toBe() uses assertEquals() for strings and assertMatchesArray() for arrays under the hood.

Custom Judge Classes

Use assertPasses() to plug in your own Judge implementation (see Custom Judges):

There is no BDD alias for assertPasses().

Judge Result Inspection

Use the judge() method to run a judge and get back a JudgeResult object with passed, score, and reasoning:

Judge Provider Override

By default, judges use the provider and model from your config/evals.php. Override per-test with judgeWith():

Custom Judge Instructions

Append extra instructions to the built-in judge prompt with judgeInstructions():

Deterministic Assertions

These are classic PHP checks — no LLM involved. They are fast, free, and predictable. There are no BDD aliases for these methods, but they chain freely with BDD methods.

String Assertions

Length Assertions

JSON Assertions

Type Assertions

Equality Assertions

Mixing BDD and Deterministic

You can freely combine them:

Tool Assertions

For agents that call tools (e.g., web search, database lookup). These check which tools were called and with what arguments.

Tools can be referenced by class (recommended, type-safe) or by string name:

Checking Tool Arguments

Pass an array for exact argument matching, or a closure for flexible inspection:

The closure receives a ToolInvocation object. You can access tool arguments directly as properties (e.g. $tool->query) thanks to magic __get. The assertion passes when at least one invocation satisfies the closure.

Asserting a Tool Was Not Used

Tool Call Sequence

Check that tools were called in a specific order (other tools may appear between them):

Tool Call Counts

Count methods also accept an optional closure as the last argument — only invocations matching the closure are counted:

ToolInvocation Properties

When inspecting tool calls via closures, the ToolInvocation object provides:

Structured Output Assertions

For agents that return arrays or objects instead of plain text (agents implementing HasStructuredOutput).

The BDD method toBe() handles exact matching. For more granular checks, use the assert* methods:

Check Keys Exist

assertHasProperty() and assertHasProperties() are aliases for assertHasKey() and assertHasKeys().

Partial Array Match

This checks that the output contains at least these key-value pairs. Extra keys are allowed.

Using Pest's expect() Directly

Call ->run() to get the raw EvalResult and use Pest's native expectations for anything not covered:

The EvalResult object gives you full access to the agent's response:

Sampling

LLMs are non-deterministic — the same prompt can produce different outputs each time. A single lucky run doesn't prove your agent is reliable. Sampling runs the agent multiple times with the same input and checks every output, giving you confidence that performance is consistent.

Basic Sampling

Chain ->samples() to run the agent N times. All samples must pass every assertion:

This runs the agent 5 times. If even one sample fails, the test fails.

Allowing Some Variance

LLMs aren't perfect. If you're OK with occasional misses, set a minimum:

repeat() Alias

repeat() is an alias for samples() — use whichever reads better:

Sampling with Scored Assertions

Each sample is scored independently and must individually meet the threshold:

Sampling with Deterministic Assertions

Every assertion type works with sampling. Each sample is checked individually:

Sampling with Tool Assertions

Tool assertions under sampling check each sample independently:

Accessing Sample Results

Call ->run() with sampling to get a SampleResults collection:

You can also judge the samples manually and inspect aggregate results:

Sampling with Datasets

Sampling composes naturally with Pest datasets — each case runs N times:

Datasets

When you have multiple test cases for the same agent, datasets keep things organized. You can define cases inline, load them from JSON or XML files, or auto-discover them from a directory.

Inline EvalCase

Create cases with EvalCase::make():

Only prompt is required. expected and attachments are optional:

JSON Datasets

JSON datasets contain one case per file. Use the .case.json extension for auto-discovery.

Only "prompt" is required. "expected" can be a string, object, or omitted. "attachments" is optional:

Attachment types: "document" or "image". Source types: "storage" (Laravel storage disk) or "path" (absolute filesystem path).

Load a single JSON file:

XML Datasets

XML datasets support multiple cases per file using an <evalset> container. Use the .case.xml extension.

When <expected> contains child elements, it's deserialized as an associative array (for structured output):

XML cases can also have attachments:

Load cases from an XML file:

Directory Auto-Discovery

EvalCase::fromDirectory() scans a directory for all *.case.json and *.case.xml files and returns them as a keyed array:

Recommended Directory Structure

Loading Methods Summary

Custom Judges

For simple criteria, a plain string works fine: ->toMeet('The tone is professional'). When you need reusable, structured evaluation logic, create a Rubric or Judge class.

Rubric Classes

A Rubric defines evaluation criteria as a reusable class. Extend Redberry\Evals\Contracts\Rubric and implement description():

Use it with toMeet() (or assertMeets()):

Judge Classes

For complete control over evaluation logic, implement the Redberry\Evals\Contracts\Judge interface. The evaluate() method receives an EvalContext and must return a JudgeResult:

Use it with assertPasses():

Custom Judge Instructions

For quick, one-off customization without creating a class, use judgeInstructions() to append extra context to the built-in LLM judge prompt:

Configuration

Config File

Publish and edit config/evals.php:

Environment Variables

Set these in .env.testing:

Per-Test Overrides

Override the agent's provider/model via prompt() or fluent methods (see Sending Prompts). Override the judge's provider/model with judgeWith():

CLI Output

Standard Output

Evals integrate with Pest's standard output:

Verbose Output

Enable verbose mode to see input, output, judge reasoning, and scores for every assertion. Turn it on with the --evals-verbose CLI flag or by setting EVALS_VERBOSE=true:

Verbose output for a failed test looks like:

Sampling Output

When using ->samples(), verbose mode shows per-sample results:

Running in CI/CD

Evals make real API calls, which means they are slow, cost money, and require API keys. You'll usually want to skip them in CI pipelines and run them manually or on a schedule instead.

Option 1: Pest Groups (Recommended)

Assign your evals to a Pest group, then exclude that group in CI.

Tag your eval tests with the evals group:

You can tag an entire file at once by adding this at the top:

Then exclude the group in your CI pipeline:

Or add a dedicated composer script in composer.json:

Now composer test skips evals, and composer test:evals runs only evals.

Option 2: skipOnCi()

If you prefer not to manage groups, use Pest's built-in skipOnCi() method on individual tests:

This skips the test whenever the CI environment variable is set (which GitHub Actions, GitLab CI, and most CI providers set automatically).

Full Examples

Basic Agent Evaluation

Structured Output Agent

Agent with Tools

Dataset-Driven Evaluation with Sampling

Complete Test Suite with Rubrics and Datasets

API Reference

Entry Point

Prompting & Configuration

LLM-as-a-Judge Assertions

Deterministic Assertions

Tool Assertions

Structured Output Assertions

Sampling

SampleResults (returned by ->run() or ->judge() when sampling)

Datasets

Judge Configuration

Execution