Download the PHP package brynj-digital/laravel-scout-vectorize without Composer

Cloudflare Vectorize Driver for Laravel Scout

A Laravel Scout driver for Cloudflare Vectorize, enabling semantic search using vector embeddings in your Laravel applications.

Features

• Semantic Search: Search by meaning, not just keywords
• Native Scout Integration: Works seamlessly with Laravel Scout
• Cloudflare Workers AI: Automatic embedding generation using Cloudflare's AI models
• Easy Setup: Simple configuration and migration from other Scout drivers
• Batch Operations: Efficient bulk indexing and deletion
• Multiple Models: Support for searching across different Eloquent models

Requirements

• PHP 8.1 or higher
• Laravel 10.x, 11.x, or 12.x
• Laravel Scout 10.x or 11.x
• A Cloudflare account with Vectorize enabled
• Cloudflare API token with Vectorize permissions

Installation

Install the package via Composer:

Publish the configuration file:

Configuration

1. Create a Vectorize Index

Use the provided artisan command to create a Vectorize index:

Alternative: Using Wrangler CLI

The dimensions must match your chosen embedding model:

• @cf/baai/bge-small-en-v1.5: 384 dimensions
• @cf/baai/bge-base-en-v1.5: 768 dimensions (default)
• @cf/baai/bge-large-en-v1.5: 1024 dimensions

2. Create Metadata Indexes

Create metadata indexes to enable efficient filtering using the artisan commands:

Note: Recent versions of this package no longer require a key metadata index, as model keys are now extracted directly from the vector ID format. This provides cleaner metadata and reduced storage requirements.

Optional: Additional Metadata Indexes for where() Clauses

You can create additional metadata indexes for any custom fields you want to filter on using Scout's where() method:

Alternative: Using Wrangler CLI

Managing Metadata Indexes

Use the provided commands to manage your metadata indexes:

To use these filters, include the fields in your model's toSearchableArray():

Then use where() in your searches:

How it works: All fields from toSearchableArray() are:

1. Converted to text and used to generate the embedding vector for semantic search
2. Stored as metadata for filtering with where() clauses

This means you can search semantically while also applying exact-match filters.

3. Create API Token

You'll need a Cloudflare API token with Vectorize permissions to allow Laravel to interact with your Vectorize index.

Create the token in Cloudflare Dashboard:

1. Log in to your Cloudflare Dashboard
2. Navigate to My Profile (click your user icon in the top right)
3. Select API Tokens from the left sidebar
4. Click Create Token
5. Choose Create Custom Token
6. Configure your token:
   • Token name: Give it a descriptive name (e.g., "Laravel Scout Vectorize")
   • Permissions: Add the following two permissions:
     • Account → Vectorize → Read
     • Account → Vectorize → Write
   • Account Resources: Select your specific account (or "All accounts" if needed)
   • TTL: Set an expiration date or leave as default
7. Click Continue to summary
8. Review the permissions and click Create Token
9. Important: Copy the token immediately - it will only be shown once
10. Store the token securely (you'll add it to your .env file in the next step)

Token Permissions Summary

Your token must have these permissions:

• ✅ Vectorize Read - Allows reading from your Vectorize indexes
• ✅ Vectorize Write - Allows creating, updating, and deleting vectors

Security Note: Avoid using tokens with broader permissions (like "Account Settings: Read" or "Workers: Edit") unless absolutely necessary.

4. Environment Variables

Add the following to your .env file:

5. Scout Configuration

Ensure Scout is configured in config/scout.php:

Usage

Basic Model Setup

Add the Searchable trait to your model:

Custom Text Conversion (Optional)

For more control over how your model is converted to searchable text, implement a toSearchableText() method:

Searching

Indexing

Removing from Index

Model Observers

Scout automatically syncs your models when you create, update, or delete them:

Practical Examples

E-commerce Product Search

Blog Article Search

Documentation Search

Customer Support Ticket Search

Advanced Usage

Custom Search Callbacks

For advanced search requirements, use a callback:

Using Where Clauses for Filtering

You can combine semantic search with metadata filtering:

Note: Filters are applied to metadata stored in Vectorize. Make sure the fields you filter on are:

1. Included in your model's toSearchableArray()
2. Have corresponding metadata indexes created in Vectorize (see Configuration section)

Querying the Client Directly

Queueing Scout Operations

For better performance in production, queue your Scout operations:

This will queue all indexing operations, preventing API rate limits and improving response times.

Available Commands

This package provides custom commands for managing Vectorize indexes and metadata indexes, plus the standard Laravel Scout commands:

Vectorize Index Management

Options for vectorize:create-index:

• name (optional): Index name (uses config value if not provided)
• --dimensions: Vector dimensions (default: 768)
• --metric: Distance metric - cosine, euclidean, or dotproduct (default: cosine)
• --embedding-model: Cloudflare embedding model (default: @cf/baai/bge-base-en-v1.5)

Options for vectorize:drop-index:

• name (optional): Index name (uses config value if not provided)
• --force: Skip confirmation prompts

Metadata Index Management

Arguments for vectorize:create-metadata-index:

• property-name: The metadata property to index
• type: Property type (string, number, boolean)

Arguments for vectorize:delete-metadata-index:

• property-name: The metadata property to delete

Options for metadata index commands:

• --index-name: Vectorize index name (uses config value if not provided)
• --force: Skip confirmation prompts (delete command only)

Standard Scout Commands

How It Works

1. Indexing: When a model is indexed, the driver:

   • Calls toSearchableText() or flattens toSearchableArray() to text
   • Generates an embedding using Cloudflare Workers AI
   • Stores the vector in Cloudflare Vectorize with metadata

2. Searching: When you search:

   • Your query text is converted to an embedding
   • Vectorize finds the most similar vectors
   • Results are mapped back to your Eloquent models
   • Models are fetched from your database and returned
3. Vector IDs: The driver prefixes vector IDs with the model class name to support multiple model types in one index (e.g., App_Models_Product_123)

Limitations

• No traditional filters: Vector search doesn't support WHERE clauses like traditional search engines. Apply filters in PHP after retrieval or use metadata filtering (which may not work reliably in all cases)
• No offset-based pagination: Vector search returns top-K results. Use cursor-based pagination or retrieve more results upfront
• Metadata filtering: Cloudflare Vectorize metadata filtering may not be reliable for all use cases. Consider filtering in your application layer
• Eventual consistency: There may be a slight delay between indexing/deletion and seeing changes in search results

Configuration Reference

Troubleshooting

Search returns no results

• Ensure your models are indexed: Run php artisan scout:import "App\Models\Product"
• Check your Vectorize index has vectors: Use the Cloudflare dashboard or API to verify
• Verify your API credentials: Double-check CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN in your .env
• Check model filters: The driver automatically filters by model class. Ensure you're searching the right model

Indexing is slow

• API overhead: Vector embedding generation requires API calls to Cloudflare Workers AI
• Use batch operations: Use makeAllSearchable() for bulk indexing (more efficient than individual saves)
• Enable queuing: Set 'queue' => true in config/scout.php to process indexing in the background
• Rate limits: Cloudflare has rate limits on API calls. Implement throttling or use queues

Errors about dimensions

• Dimension mismatch: Ensure your Vectorize index dimensions match your embedding model
  • @cf/baai/bge-small-en-v1.5: 384 dimensions
  • @cf/baai/bge-base-en-v1.5: 768 dimensions (default)
  • @cf/baai/bge-large-en-v1.5: 1024 dimensions
• Recreate index: If you changed embedding models, you'll need to create a new index with the correct dimensions

Authentication errors

• Invalid API token: Verify your CLOUDFLARE_API_TOKEN has Vectorize permissions
• Incorrect account ID: Double-check your CLOUDFLARE_ACCOUNT_ID
• Token permissions: Ensure your API token has Vectorize read and write permissions

Metadata filtering not working

• Create metadata indexes: Metadata filters require indexes. Run:

• Check field types: Ensure the metadata index type matches your data (string, number, boolean)
• Include in searchable array: The field must be in your model's toSearchableArray()

Performance optimization

• Limit result size: Use take() or paginate() to limit results
• Cache frequent queries: Cache search results for common queries
• Use metadata filters wisely: Filters can reduce the search space and improve performance
• Optimize text conversion: Keep toSearchableText() concise to reduce embedding generation time

Architecture

Package Structure

How Embeddings Work

This package uses Cloudflare Workers AI to generate embeddings:

1. Text Preparation: Your model data is converted to text using toSearchableText() or by flattening toSearchableArray()
2. Embedding Generation: The text is sent to Cloudflare Workers AI which returns a vector (array of floats)
3. Vector Storage: The vector is stored in Vectorize along with metadata (model class and searchable data)
4. Semantic Search: When you search, your query is also converted to a vector and compared against stored vectors using cosine similarity

Supported Embedding Models

Vector ID Format

Vectors are stored with IDs in the format: {ModelClass}_{ModelKey}

Example: App_Models_Product_123

This allows multiple model types to coexist in the same Vectorize index.

Testing

The package includes comprehensive tests covering all engine functionality:

Test Coverage

The test suite includes 23+ tests covering:

• Update operations: Empty collections, valid models, custom text conversion, array values
• Delete operations: Empty collections, model deletion
• Search operations: Default limits, custom limits, filters, callbacks, pagination
• Result mapping: ID extraction, model mapping, ordering
• Flush operations: Batch deletion, different embedding models
• Index operations: Create/delete (no-op for Vectorize)

Running Tests

Tests use Orchestra Testbench to simulate a Laravel environment and Mockery to mock the VectorizeClient, ensuring tests run without making actual API calls.

Best Practices

Optimizing Search Quality

1. Use descriptive text: Include context in your searchable content

2. Avoid overly long text: Embeddings work best with focused, relevant content

3. Include relevant metadata: Add fields you'll filter on

Performance Tips

1. Enable queueing for production: Prevent blocking requests

2. Use batch operations: Import in bulk rather than one-by-one

3. Limit search results: Only fetch what you need

4. Cache frequent queries: Use Laravel's cache for popular searches

Security Considerations

1. Sanitize user input: Always validate and sanitize search queries

2. Protect API credentials: Never commit API tokens to version control

3. Use scopes for access control: Filter by user permissions

Comparison with Other Search Solutions

When to Use Vectorize

Good fit:

• Semantic/conceptual search (finding by meaning, not keywords)
• Multi-language search (embeddings understand concepts across languages)
• Finding similar content or recommendations
• Applications already using Cloudflare
• Budget-conscious projects needing semantic search

Not ideal for:

• Exact keyword matching
• Complex filtering and faceting requirements
• Typo-tolerant search
• Traditional full-text search
• Applications requiring instant consistency

FAQ

Q: Can I use multiple models in the same index? A: Yes! The driver automatically namespaces vectors by model class, so multiple models can coexist in one index.

Q: How accurate is semantic search compared to keyword search? A: Semantic search excels at understanding intent and meaning, but may miss exact keyword matches. Consider your use case.

Q: Can I migrate from Algolia/Meilisearch to Vectorize? A: Yes, but be aware that Vectorize uses semantic search, which behaves differently from keyword-based search engines.

Q: What happens if I change the embedding model? A: You'll need to create a new index with the correct dimensions and re-index all your data.

Q: Is there a limit on the number of vectors? A: Check Cloudflare's Vectorize pricing and limits for your account tier.

Q: Can I use this with multilingual content? A: Yes! The BGE embedding models support multiple languages and can find semantically similar content across languages.

Contributing

Contributions are welcome! Please submit pull requests or open issues on GitHub.

Development Setup

License

This package is open-source software licensed under the MIT license.

Credits

• Built for use with Cloudflare Vectorize
• Integrates with Laravel Scout

Support

For issues, questions, or contributions, please visit the GitHub repository.