Laravel Realtime Agent

Provider-agnostic realtime agents with Laravel-authoritative state, goals, tools, and semantic UI control.

Laravel Realtime Agent is an installable Composer package for Laravel 12 and 13. It keeps audio and realtime events close to the provider while every consequential operation—state mutation, goal completion, tool execution, authorization, confirmation, UI action, transcript, and usage record—passes through Laravel.

The package ships a private, framework-neutral TypeScript runtime in the same Composer distribution. The default provider is deterministic and free: no API key is required to develop or run the standard test suite.

Why this package exists

A conversational provider is good at low-latency voice. It should not be the authority for your application.

This package separates the two concerns:

• the provider carries realtime media and translates vendor events;
• Laravel owns the canonical snapshot, revision checks, policies, tool broker, goals, ordered transcript, cost audit, and signed UI commands;
• the browser exposes only a semantic Surface, never a raw DOM dump or arbitrary selectors;
• OpenAI and ElevenLabs IDs remain inside their adapters, outside application code.

Requirements

• PHP 8.2 or newer
• Laravel 12 or 13
• Node.js 20 or newer when bundling the browser client
• a database supported by Laravel
• HTTPS for browser microphone access

Install in a Laravel application

Install the stable 0.1 line from Packagist:

When developing against this local checkout instead of a tagged release:

Set the application to the Fake Provider in .env:

Open the project as realtime-agent-demo.test with Laravel Herd and enable HTTPS for that site. Do not add OpenAI or ElevenLabs credentials for the standard test path.

The install command publishes:

• config/realtime-agent.php;
• six timestamped migrations, without duplicating an already installed migration;
• the compiled browser runtime under public/vendor/realtime-agent.

Use --force only when you intentionally want to refresh published configuration, migrations, and assets.

Handoff

This repository ships an installable Codex skill containing the complete integration manual. After requiring the package in a target Laravel application, install or refresh the skill with:

Open a new Codex task and invoke $install-laravel-realtime-agent. During local package development, the source skill is available at skills/install-laravel-realtime-agent and can be copied directly from this checkout.

Package maintainers can also install the release-review skill:

Invoke $release-laravel-realtime-agent to audit a candidate, run the complete offline-safe gate, prepare SemVer and changelog metadata, create the atomic local release commit, and add an annotated v<version> tag. The skill never pushes, publishes, or runs paid provider tests without a separate explicit request.

Copy the following prompt into AskMyDoc or any other Laravel project when handing the integration to another agent:

`

The repository-level AGENTS.md requires this Handoff and the bundled skill to be updated whenever installation, public APIs, provider behavior, security, audit, or verification changes.

Anonymous case studies

The bundled skill contains three reusable, fictionalized patterns in case-studies.md. They are not references to, documentation for, or compatibility claims about any external product:

• Adaptive-learning session — ordered goals, retrieval and quiz tools, React chat, transcript projection, and billing limits.
• Connected-environment controller — authorization, revisioned device commands, confirmations, presentation-only Surface actions, and operational audit.
• Revisioned creative workspace — method objectives, card-based UI, dual revision checks, conversation projection, and broadcast patches.

When a target resembles one of these anonymous patterns, append this sentence to the generic prompt above:

Create a session

Sessions are created by trusted application code, never by a generic browser endpoint:

startFor() returns a StartedAgentSession. It exposes the session ID, initial state, client connection descriptor, and the normal executable session methods.

Connect the browser runtime

Expose the descriptor as JSON in Blade:

Then register a semantic Surface and connect the client:

The declarative adapter reads only elements explicitly marked with data-agent-* attributes:

Programmatic registration is available through client.surface.register(...). The public client also exposes connect, disconnect, sendText, switchToText, switchToVoice, audit, refreshState, finish, and on.

Continue the same session in text

Switching channel does not finish the Laravel session or discard its canonical context. With OpenAI, switchToText() gracefully closes the billed GPT-Live voice session and subsequent messages use the configured Responses backend through Laravel. Both sides remain canonical messages in the same session:

Call switchToVoice() to create a fresh GPT-Live WebRTC connection seeded with the saved conversation, while retaining the same Laravel session and goals. Other providers may switch modality on their existing transport. Call disconnect() when provider transport should stop; call finish() when the canonical Laravel session itself is complete.

Every persisted conversation row has a per-session sequence, role, input/output direction, text/audio modality, provider event ID, status, timestamp, and metadata. GPT-Live exposes transcript fragments rather than authoritative turns: the runtime groups nearby fragments into revisable rows, keeps user and assistant timelines independent, and preserves every original delta, start_ms, and end_ms inside message metadata. This avoids thousands of partial database rows without discarding the source evidence.

Tools, confirmations, and state

Application tools use one canonical definition regardless of provider:

Built-in tools are opt-in per session:

• runtime.state.get
• working_memory.update
• goal.update
• ui.action
• session.finish

State writes use optimistic base_revision checks. A stale mutation returns HTTP 409 with the current state. Policies return 403, invalid schemas return 422, and expired UI commands return 410. Tool calls are idempotent, events are append-only, sensitive arguments are redacted from audit by default, and UI commands carry a short-lived server signature plus a one-use nonce.

Control routes are mounted under /realtime-agent with web and auth middleware by default. Both prefix and middleware are configurable. Session ownership is checked again inside the controller; an application can set security.authorization_ability to a Laravel Gate ability for custom authorization.

Transcript and cost audit

The database keeps three complementary audit streams:

• realtime_agent_messages stores finalized user and assistant text in strict session order;
• realtime_agent_usage stores normalized units, the original provider usage payload, a snapshot of the rate card used, amount, currency, and confidence status;
• realtime_agent_tool_calls stores tool lifecycle, authorization, confirmation, revisions, redacted arguments, result, and error.

Fetch the combined, versioned view from trusted PHP code:

Or, for the authorized session owner, request:

audit.totals separates estimated, provider-confirmed final, and effective totals. effective prefers a final provider amount when one exists; unpriced_records makes unsupported units visible instead of silently valuing them at zero.

GPT-Live voice usage arrives as cumulative session.usage.updated snapshots. The package updates one duration record per provider session instead of summing those snapshots, then captures the last value from session.closed. Responses-backend token usage is recorded separately from nested response.completed events. The catalog currently prices gpt-live-1 by second and gpt-5.6-terra by input, cached-input, and output tokens; each record keeps the exact rate-card snapshot used for its estimate. This follows the official GPT-Live cost guide, GPT-Live pricing, and GPT-5.6 Terra pricing.

OpenAI includes WebRTC initialization in the reported duration: the initial 15 seconds are credited against running time and must not be added a second time. Backend tools and models remain separate cost lines, so an auditor can distinguish conversation time from task execution.

ElevenLabs sends final user/agent text over the live socket. After a call is processed, reconcile it to import missed turns and the provider-reported cost_fiat:

That operation runs server-side, requires the configured ElevenLabs key, and uses the recorded conversation ID. See the official conversation details response.

Provider session IDs are accepted only from trusted server bootstrap responses or application PHP code; the browser cannot replace the ID used for reconciliation.

External accounting, observability, or data-warehouse code can intercept records without coupling to a provider adapter:

For batch audits across many sessions, AgentSessionRecord exposes messages, usageRecords, and toolCalls Eloquent relations, while provider_session_id links each Laravel session to the provider conversation or call.

OpenAI browser-originated usage remains explicitly estimated, because WebRTC server events reach the browser. A trusted billing webhook or reconciliation job can call SessionAuditManager::recordProviderCost(...) to add a provider-confirmed final amount. ElevenLabs reconciliation does this automatically from its authenticated server API. Browser requests cannot supply a monetary amount.

Providers

Fake

fake is the default. It performs no external HTTP calls and exposes deterministic provider events for unit, feature, and browser tests.

OpenAI GPT-Live

Set these only when deliberately testing the live adapter:

After adding the key locally, verify the resolved configuration without contacting OpenAI:

Laravel sends the browser SDP offer to POST /v1/live/sessions; the standard API key never leaves the server. Audio uses WebRTC while the oai-events data channel carries Live events. The package waits for session.started, uses Responses delegation for gpt-5.6-terra, reads function calls from nested response.output_item.done, and returns authorized results with response.item.create followed by response.create. Application tools still pass through the Laravel Tool Broker.

Typed values can be supplied during a voice call through Live Responses delegation. A full switchToText() closes the Live transport first, preventing silent voice-duration billing, then routes written messages and authorized tool results to /v1/responses through Laravel. Switching back to voice creates a new Live transport from canonical history. Surface changes use bounded session.thinking.append context, and every voice close waits for session.closed before releasing WebRTC so the final transcript and duration events can drain.

When a browser establishes a new provider connection for an existing Laravel session, recent canonical user/assistant messages are supplied as GPT-Live startup history. OPENAI_LIVE_STORE is off by default to minimize provider-side recording; enable it only when your data policy permits stored recordings and future provider-side forks. See the official GPT-Live WebRTC guide, session guide, and delegation guide.

ElevenLabs

Laravel returns a signed WebSocket URL. Canonical tool schemas are materialized as ElevenLabs client tools and cached by schema hash in realtime_agent_provider_tools. Surface updates use contextual updates, and client tool calls always return through Laravel. See the official signed URL and client event references.

No live provider test runs in the standard suite or in CI. GPT-Live has no free-tier access; the default Fake Provider and all OpenAI HTTP/WebRTC contract tests need no credential.

Test the package itself

From this repository:

The commands cover:

• PHPUnit and Orchestra Testbench for state revisions, database persistence, goals, ownership, authorization boundaries, confirmations, signed UI commands, ordered transcripts, provider cost normalization/reconciliation, and idempotency;
• Pint and Larastan for PHP style and static analysis;
• TypeScript build, JSON Schema synchronization, Vitest, and jsdom for Surfaces and the command executor;
• the README validator for local image existence, raster integrity and size, SVG accessibility metadata, alt text, and the required Handoff section;
• Playwright for the full Fake Provider scenario: three goals, three UI updates, voice-to-text continuation, then a completed session.

You can inspect configuration readiness without contacting a provider:

With the Fake Provider, the expected result is ready (no credentials required).

Test in a real Laravel app

After the local installation above:

1. Create a controller action that starts a Fake Provider session with three declared goals.
2. Pass $started->connection() to a Blade page.
3. Add a CSRF meta tag and at least one data-agent-id element.
4. Instantiate FakeRealtimeDriver, LaravelControlTransport, and RealtimeAgentClient as shown above.
5. Open https://realtime-agent-demo.test through Herd.
6. Verify php artisan realtime-agent:doctor, then inspect the Network panel: control calls stay under /realtime-agent, and no provider domain is contacted.

For a fully deterministic reference, open examples/fake-lesson.html through the Playwright test server by running npm run test:e2e.

Architecture map

The database is the only durable state store in v0.1. Redis, Reverb, server-side sideband connections, and dedicated React/Vue/Livewire adapters are intentionally deferred; the core browser API already works with those frameworks through programmatic Surface registration.

Security model

• Browser, provider, route data, Surface snapshots, and tool arguments are untrusted.
• Only registered tool names, semantic targets, and action names are accepted.
• Raw CSS selectors, HTML, scripts, and JavaScript fields are rejected from Surface input.
• Authorization runs at session and tool level.
• UI commands expire, are signed with APP_KEY, and are removed after completion.
• Provider keys stay on Laravel; they are never serialized into the client descriptor.
• Current state is compact; events, finalized messages, usage, and tool calls remain durable for audit and debugging.
• Browser telemetry cannot set monetary amounts; only configured server pricing or trusted reconciliation can do so.

Version scope

The current stable line is 0.1.x, beginning with v0.1.0. See LICENSE.md for licensing.