Download the PHP package casawatt/laravel-ai-agent-evaluation without Composer

Laravel AI Agent Evaluation

Evaluate your Laravel AI SDK agents across multiple providers and models. Compare performance, accuracy, and cost.

Installation

Publish the config file:

Quick Start

1. Create an evaluation

This creates:

2. Define your evaluation

Edit agent-evaluations/SalesCoachEvaluation.php:

3. Run evaluations

Each case is run against every variant. Results are displayed in the console and persisted to SQLite.

Variants

Variants define the configurations to test against. Configure them in setUp():

Each variant can have a custom label (used in the results table) and a custom instruction (overrides the agent's default system prompt):

Instructions can also be loaded from a file using the file:// prefix — useful for long or complex system prompts:

Relative paths are resolved from the evaluations directory (agent-evaluations/ by default). Absolute paths are used as-is.

This lets you compare the same model with different instructions — useful for prompt engineering and evaluating system prompt variations.

Generation Options

Variants can override the model's generation parameters — handy for comparing the same model at different temperatures or token budgets:

The four options map directly to the Laravel AI SDK's generation parameters:

Any option you leave unset falls back to the value defined on the agent itself (its temperature() method or #[Temperature] attribute, and so on). When at least one variant sets a generation option, the variant summary table shows a Params column (e.g. temp=0.0 maxTok=512), and each cell in the HTML report lists the active parameters.

Cost Tracking

Add pricing to variants to track cost per evaluation. Pricing is defined in dollars per million tokens:

When at least one variant has pricing, the variant summary table shows an Avg Cost column (average cost per case). Variants without pricing show —.

Cost Resolvers

Instead of setting pricing on each variant, you can register cost resolvers that automatically look up pricing by provider and model. Resolvers are tried in order — the first non-null result wins. Explicit pricing() on a variant always takes precedence.

The package ships with two built-in resolvers:

Enable them in config/ai-agent-evaluation.php:

Each API is called once per run and cached in memory. Resolvers are tried in order — place more specific resolvers first.

You can create your own resolvers by implementing CostResolverInterface:

Custom providers (not in the Lab enum) work as long as they are configured in your config/ai.php.

Assertions

All assertions are chainable. The package returns an AssertableResponse for text agents and an AssertableStructuredResponse for agents implementing HasStructuredOutput — the correct type is detected automatically.

Text (AssertableResponse)

Structured Data (AssertableStructuredResponse)

Available automatically when your agent implements HasStructuredOutput. These work directly on the parsed $response->structured array — no JSON parsing needed.

All text assertions are also available, with a string $path (dot-notation) as the first argument — the assertion runs against the value at that path:

Plus structure-specific assertions:

Tool Calls

Performance

Custom

Weighted Assertions

Every assertion has a weight parameter (default 1.0). Assertions never throw — failures are recorded and execution continues, so all assertions in a case always run. The weight is a float between 0 and 1 that indicates the assertion's importance:

• 1.0 — full importance (default)
• 0.5 — half as important
• 0.1 — nice-to-have

The variant summary table shows a Score column with weighted percentages:

Metrics

Every assertion accepts an optional metric tag to group assertions by quality dimension (e.g. accuracy, completeness, safety). Metrics aggregate scores across all cases within a suite, giving you a per-dimension breakdown by variant.

When any assertion has a metric, the console output includes an additional Metrics table:

Metrics are persisted to storage alongside each assertion result, so they are available for web reporting without re-running evaluations.

Data Providers

Use #[With('methodName')] to feed multiple data sets into a single case — like PHPUnit's #[DataProvider]. The method can load data from anywhere: arrays, models, files, APIs.

Each data set becomes a separate row in the results: knows_capital (france), knows_capital (germany), etc. The keys are used as labels.

The provider method can return any data — query a database, read a CSV, call an API:

Skipping Cases

Some providers or models may not support certain features (e.g. tool use). You can skip cases conditionally using skip() or skipWhen():

skipWhen() accepts a boolean or a callable that receives the current Variant. You can also call skip() to unconditionally skip:

Skipped cases show S in the progress output and SKIP in the test matrix. They do not count as failures.

Parallel Execution

By default, cases run sequentially. Use --parallel to run multiple cases concurrently using spatie/fork (requires the pcntl extension):

Each case runs in its own forked process with a fresh evaluation instance — no shared state. Results are persisted to storage by the parent process after each child completes.

You can set a default in config/ai-agent-evaluation.php:

The --parallel CLI option overrides the config value. Set to 1 for sequential execution.

Command Options

Storage

Results are persisted to a SQLite database after each case completes — if the process is killed, all completed results are preserved. The database is stored at storage/ai-agent-evaluation/evaluations.sqlite by default.

The package uses raw PDO with WAL mode — no Laravel database connection required. Safe for concurrent writes.

Resume and Retry

--resume — Loads the latest run from storage and skips all case+variant combos that already have results. Only missing cases are executed. Useful when a run was interrupted.

--retry-errors — Like resume, but re-executes cases with error status (API failures, timeouts). Passed/failed/skipped results are kept as-is.

--retry-failed — Like resume, but re-executes cases with failed status (assertions that didn't pass). Passed/errored/skipped results are kept as-is.

Each result has a status: passed, failed, skipped, or error. Assertion failures are failed; exceptions are error.

Results

Console output includes up to three tables:

Test Matrix — each case x each variant with PASS/FAIL/ERROR/SKIP, latency, and token count.

Variant Summary — aggregated pass rate, average latency, total tokens, and cost per variant.

Metrics (when assertions use metric:) — per-metric score breakdown by variant.

Testing

Credits

• Olivier Pousset
• All Contributors

License

The MIT License (MIT). Please see License File for more information.