Download the PHP package shamrozghouri/laravel-agent-evals without Composer

Laravel Agent Evals

Define AI-agent behavior as test cases, run them against your real agent, and automatically catch regressions before they ship.

Table of Contents

• What is this?
• Why use it?
• How it works
• Requirements
• Installation
• Quick Start (5 minutes)
• Connecting Your Agent
• Writing Test Cases
  • Assertions
  • Tags
• Running Evals
• Baselines & Regressions
• Continuous Integration (CI)
• Optional Dashboard
• Configuration Reference
• Troubleshooting
• Reporting a Problem
• Contributing
• License

What is this?

Laravel Agent Evals is a testing tool for AI agents built with Laravel. Instead of manually checking whether your chatbot, assistant, or LLM-powered feature still behaves correctly after every change, you define its expected behavior as test cases and let the package verify them for you.

Think of it as PHPUnit for AI behavior: you describe what the agent should (or shouldn't) do, run a command, and get pass/fail results.

Why use it?

LLM-powered agents break silently. A tweaked prompt, a model upgrade, or a new tool can quietly change how your agent responds — and you often won't notice until a user complains.

This package helps you:

• 🧪 Turn behavior into tests — version-controlled and repeatable.
• 🚨 Catch regressions automatically — get alerted when something that used to work stops working.
• 🏷️ Focus your testing — run only the cases you care about (e.g. safety, billing).
• 🤖 Fit into CI — fail a pull request automatically when the agent misbehaves.

How it works

1. You write test cases describing inputs and expected behavior.
2. The package sends each input to your agent.
3. It checks the response against your assertions.
4. Results are compared against a saved baseline to detect regressions.

Requirements

Installation

Install via Composer:

Publish the configuration file:

This creates config/agent-evals.php.

Quick Start (5 minutes)

1. Tell the package about your agent in config/agent-evals.php:

2. Create your first test case in tests/AgentEvals/greeting.php:

3. Run it:

You'll see a pass/fail result in your terminal. That's it — you now have your first automated agent eval. 🎉

Connecting Your Agent

The package calls your agent through Laravel's service container, so any class that's resolvable by the container will work.

Your agent method must:

• accept a single string input, and
• return a string response.

Register it in config/agent-evals.php:

Writing Test Cases

Test cases live in tests/AgentEvals. Each file must return a single TestCase or an array of them.

Assertions

Custom rule example:

Tags

Tags let you group and target related cases (e.g. by topic or risk area):

You can then run only those cases — see Running Evals.

Running Evals

Baselines & Regressions

The first successful run saves a baseline — a snapshot of known-good results — to:

On later runs, the package compares against this baseline:

• ✅ A case that keeps passing → no problem.
• 🚨 A case that previously passed but now fails → reported as a regression, and the command exits with a non-zero code (which fails CI).

Important: Tagged runs (--tag=...) never update the baseline. This prevents a partial run from overwriting your full suite's known-good snapshot.

Continuous Integration (CI)

Because regressions cause a non-zero exit code, wiring this into CI takes almost no effort. Example for GitHub Actions (.github/workflows/agent-evals.yml):

Optional Dashboard

A simple web dashboard lets you view the latest baseline in your browser and (for authorized users) trigger a full run.

It is disabled by default. To enable it, add to your .env:

Then visit /agent-evals (or your custom path).

⚠️ Security warning: Before enabling the dashboard in production, protect it with authentication by adding your middleware to dashboard.middleware in config/agent-evals.php. Never expose the ability to run evals publicly.

Configuration Reference

All options live in config/agent-evals.php:

The exact keys may vary slightly — check your published config file for the authoritative list.

Troubleshooting

Class "...TestCase" not found Make sure your use statement matches the package namespace exactly: use ShamrozGhouri\LaravelAgentEvals\TestCase;. Then run composer dump-autoload.

"No test cases found" Confirm your files are in tests/AgentEvals (or your configured path) and that each file returns a TestCase or an array of them.

"Agent class not resolvable" Ensure agent.class is correct and the class can be built by Laravel's container (its constructor dependencies are bindable).

Baseline never updates Remember: tagged runs don't update the baseline. Run the full suite (no --tag) to refresh it.

Reporting a Problem

When opening an issue, please include the following (it helps us help you fast):

Run these and paste the output:

Also include:

• Laravel version
• PHP version
• Package version
• The Artisan command you ran
• The complete error message
• A minimal example of the eval that fails

Please do NOT include:

• ❌ API keys
• ❌ Passwords
• ❌ .env contents containing secrets
• ❌ Private customer data
• ❌ Production credentials

Contributing

Contributions, issues, and feature requests are welcome! Feel free to open an issue or submit a pull request.

License

Laravel Agent Evals is open-sourced software licensed under the MIT License.