Download the PHP package mubseoul/laravel-llm-observability without Composer

Laravel LLM Observability

A Laravel-native observability and cost tracking toolkit for LLM calls (OpenAI, Claude, Ollama, and more). Track every LLM request in your Laravel application with detailed metrics, enforce quotas, monitor costs, and gain insights through a beautiful Filament dashboard.

Features

✨ Comprehensive Tracking

• Track tokens, latency, provider/model, cost estimates, and success/failure status
• User and team attribution for multi-tenant applications
• Detailed metadata capture (environment, route, IP, user agent)
• Optional storage of request/response bodies (with privacy controls)

💰 Cost Management

• Built-in pricing tables for OpenAI, Anthropic, and other providers
• Automatic cost calculation based on token usage
• Custom pricing providers for specialized models
• Cost aggregation by user, team, time period, or globally

🚦 Quota Enforcement

• Request, token, and cost limits (per day/month)
• Scoped quotas (global, per-user, per-team, per-API-key)
• Automatic quota resets via scheduled jobs
• Middleware for blocking requests when quotas exceeded

📊 Filament Dashboard

• Overview page with key metrics (requests, costs, latency, error rates)
• Detailed request logs with filtering and search
• Usage analytics by provider, model, user, and team
• Quota management interface
• Alert rule configuration

🔔 Alerts & Webhooks

• Configurable alert rules (cost thresholds, error rates, latency spikes)
• Webhook delivery with retry logic
• Laravel Notifications integration (mail, Slack, etc.)
• Audit trail for webhook deliveries

⚡ Performance

• Async recording via queue jobs
• Sampling support for high-volume applications
• Efficient data aggregation
• Configurable data retention

🔒 Privacy & Security

• Optional PII redaction
• Configurable sensitive data patterns
• Body storage disabled by default
• Secure webhook delivery

Requirements

• PHP 8.2 or higher
• Laravel 10.x or 11.x
• (Optional) Filament 3.2+ for dashboard UI

Installation

Install the package via Composer:

Publish the configuration file:

Run the migrations:

(Optional) If using Filament dashboard, install Filament:

Configuration

The config file config/llm-observability.php provides extensive customization options:

Recording Settings

Pricing Configuration

Update pricing as providers change their rates:

Prices are in USD per 1M tokens.

Quota Defaults

Usage

Basic Recording

With User Context

With Metadata

Manual Recording (without callback)

Enforcing Quotas (Middleware)

Add the middleware to your routes:

When a quota is exceeded, the middleware returns a 429 response:

Programmatic Quota Checking

Managing Quotas

Creating Alert Rules

Artisan Commands

Prune Old Logs

Recalculate Aggregates

View Pricing Table

Dashboard

The Filament dashboard provides a comprehensive UI for monitoring and managing LLM usage.

Access the dashboard at: /llm-observability (configurable)

Dashboard Features:

• Overview: Real-time metrics for the last 24h, 7d, and 30d
• Request Logs: Searchable, filterable table of all LLM requests
• Usage Analytics: Breakdown by provider, model, user, and team
• Quota Management: Configure and monitor quotas
• Alert Rules: Create and manage alert thresholds

Configure dashboard settings in config/llm-observability.php:

Scheduled Tasks

The package automatically registers these scheduled tasks:

• Daily Quota Reset: Runs at 00:01 daily
• Monthly Quota Reset: Runs on the 1st of each month at 00:05
• Log Pruning: Runs at 02:00 daily (if retention configured)

No manual scheduler configuration needed.

Integration Examples

OpenAI PHP SDK

Laravel HTTP Client

Custom Provider

Testing

Run with coverage:

Run static analysis:

Format code:

Security Considerations

Privacy

• Bodies Disabled by Default: Raw prompts and responses are NOT stored unless explicitly enabled
• Redaction Patterns: Sensitive data (API keys, passwords) is automatically redacted from metadata
• Configurable: Add custom redaction patterns in config

Recommendations

1. Review stored data: Understand what metadata is captured
2. Enable sampling: Use sampling in high-volume environments
3. Set retention limits: Configure retention.requests_days to auto-prune old data
4. Secure webhooks: Use HTTPS and validate webhook signatures
5. Audit access: Restrict dashboard access with appropriate middleware

Roadmap

• [ ] Enhanced Analytics: More dashboard widgets and charts
• [ ] Export Functionality: CSV/Excel export for finance teams
• [ ] Budget Alerts: Proactive budget threshold notifications
• [ ] Multi-tenancy: Enhanced team/organization isolation
• [ ] Rate Limiting: Built-in rate limiting by provider
• [ ] Streaming Support: Track streaming LLM responses
• [ ] Cache Integration: Track cache hits/misses for LLM responses
• [ ] Provider Auto-detection: Automatically detect provider from API calls
• [ ] Prompt Templates: Track and version prompt templates
• [ ] A/B Testing: Compare different models/prompts

Contributing

Please see CONTRIBUTING for details.

Security Vulnerabilities

If you discover a security vulnerability, please send an email to [email protected]. All security vulnerabilities will be promptly addressed.

Credits

• Mubseoul
• All Contributors

License

The MIT License (MIT). Please see License File for more information.