Download the PHP package yetidevworks/yetisearch without Composer

YetiSearch

A powerful, pure-PHP search engine library with advanced full-text search capabilities, designed for modern PHP applications.

Important: Requires SQLite FTS5 (full‑text search) support in your PHP’s SQLite library. See “Requirements” for a quick check.

Table of Contents

• Features
• Requirements
• Installation
• Quick Start
• Example Applications
• Usage Examples
  • Basic Indexing
  • Advanced Indexing
  • Search Examples
  • Document Management
• Configuration
• Advanced Features
  • Document Chunking
  • Field Boosting and Exact Match Scoring
  • Multi-language Support
  • Custom Stop Words
  • Geo-Spatial Search
  • Search Result Deduplication
  • Highlighting
  • Fuzzy Search
  • Faceted Search
• Architecture
• Testing
• API Reference
• Performance
  • Benchmark Results
  • Performance Characteristics
  • Performance Tuning
  • Bottlenecks and Solutions
  • Comparison with Other Solutions
  • Best Practices for Performance
• Future Features
• Contributing
• License
• Type-Ahead Setup
• Weighted FTS and Prefix (Optional)
• Suggestions
• Synonyms
• DSL (Domain Specific Language)
• CLI

Features

• 🔍 Full-text search powered by SQLite FTS5 with BM25 relevance scoring
• 📄 Automatic document chunking for indexing large documents
• 🎯 Smart result deduplication - shows best match per document by default
• 🌍 Multi-language support with built-in stemming for multiple languages
• ⚡ Lightning-fast indexing and searching with SQLite backend
• 🔧 Flexible architecture with interfaces for easy extension
• 📊 Advanced scoring with intelligent field boosting and exact match prioritization
• 🎨 Search highlighting with customizable tags
• 🔤 Advanced fuzzy matching with multiple algorithms (Trigram, Jaro-Winkler, Levenshtein, Basic)
• 🎯 Enhanced multi-word matching for more accurate search results
• 🏆 Smart result ranking prioritizing exact matches over fuzzy matches
• 📈 Faceted search and aggregations support
• 📍 Geo-spatial search with R-tree indexing for location-based queries
• 🚀 Zero dependencies except PHP extensions and small utility packages
• 💾 Persistent storage with automatic database management
• 🔐 Production-ready with comprehensive test coverage
• ✨ NEW: Multi-column FTS with native BM25 field weighting (enabled by default)
• ✨ NEW: Two-pass search for enhanced primary field prioritization (optional)
• ✨ NEW: Improved fuzzy consistency - exact matches always rank higher
• ✨ NEW: DSL Support - Natural language query syntax and JSON API-compliant URL parameters
• ✨ NEW: Query Result Caching - 10-100x faster repeated searches with automatic invalidation

Requirements

Important: SQLite FTS5 required

• YetiSearch uses SQLite FTS5 virtual tables for full‑text search and BM25 ranking. Your PHP build must link against a SQLite library compiled with FTS5 (ENABLE_FTS5).

• Quick check: php scripts/check_sqlite_features.php should report “FTS5: OK”. On macOS, Homebrew PHP typically includes FTS5; some system PHP builds may not.

• PHP 7.4 or higher
• SQLite3 PHP extension
• PDO PHP extension with SQLite driver
• Mbstring PHP extension
• JSON PHP extension

Installation

Install YetiSearch via Composer:

Quick Start

Example Applications

The examples/ directory contains fully working demonstrations of YetiSearch features:

🏢 Apartment Search Tutorial

Complete real-world example of a property search application:

• File: examples/apartment-search-simple.php
• Features demonstrated:
  • Structured content indexing (title, description)
  • Metadata fields (price, bedrooms, bathrooms, sqft, location)
  • Geo-spatial search with radius filtering
  • Price range and feature filtering
  • DSL queries with natural language syntax
  • Fluent query builder interface
  • Distance calculations and sorting

Run it:

🔍 Other Examples

• Pre-chunked indexing: examples/pre-chunked-indexing.php - Custom document chunking with semantic boundaries
• Type-ahead search: examples/type-ahead.php - Interactive as-you-type search
• Geo facets and k-NN: examples/geo-facets-knn.php - Distance faceting and nearest neighbors
• DSL examples: examples/dsl-examples.php - Query builder demonstrations
• Custom fields: examples/apartment-search-tutorial.php - Extended version with amenities

Usage Examples

Basic Indexing

Advanced Indexing

Search Examples

Multi-Index Search

Search across multiple indexes simultaneously:

Document Management

Configuration

Full Configuration Example

Storage Schema: External-Content (Default)

YetiSearch defaults to an efficient external-content FTS5 schema:

• Main table <index>: doc_id INTEGER PRIMARY KEY, id TEXT UNIQUE, document content and metadata, with language, type, timestamp.
• FTS5 table <index>_fts: multi-column virtual table, content='<index>', content_rowid='doc_id' (no document duplication in FTS).
• Spatial table <index>_spatial: R-tree keyed by doc_id (no separate id_map).

Legacy indices (string id primary key with id_map) continue to work. You can migrate any legacy index to the new schema:

To explicitly create an index with/without external-content:

Advanced Features

Document Chunking

YetiSearch supports both automatic and manual (pre-chunked) document chunking:

Automatic Chunking

Large documents are automatically split into smaller chunks for better search performance:

Pre-chunked Documents (Custom Chunking)

NEW: You can provide your own chunks for better semantic boundaries:

Benefits of pre-chunked documents:

• Control chunk boundaries at semantic breakpoints (paragraphs, sections)
• Preserve document structure (headings, subsections)
• Add custom metadata to each chunk
• Better search relevance by keeping related content together

See examples/pre-chunked-indexing.php for a complete example.

Field Boosting and Exact Match Scoring

YetiSearch provides intelligent field-weighted scoring with special handling for exact matches in high-priority fields:

How Field Boosting Works:

1. Basic Boost Values: Each field's boost value multiplies its relevance score

2. High-Priority Fields (boost ≥ 2.5): Get special exact match handling:

   • Exact field match: +50 point bonus (e.g., searching "Star Wars" finds a movie titled exactly "Star Wars")
   • Near-exact match: +30 point bonus (ignoring punctuation)
   • Length penalty: Shorter exact matches score higher than longer titles containing the phrase
3. Phrase Matching: Exact phrases get 15x boost over individual word matches

Example:

This intelligent scoring ensures the most relevant results appear first, with exact matches in important fields (like titles or names) getting priority over partial matches in longer text.

Enhanced Result Ranking (v1.0.3):

• Exact vs Fuzzy Priority: Regular matches always rank higher than fuzzy matches
• Shorter Match Preference: Among similar matches, shorter documents score higher
• Multi-word Query Handling: Improved matching for queries with multiple words
• Short Text Flexibility: Better handling of short text queries and matches

For more detailed information about scoring and configuration options, see the Field Boosting and Scoring Guide.

For comprehensive fuzzy search documentation, see the Fuzzy Search Guide.

Multi-language Support

Supported languages:

• English (default)
• French
• German
• Spanish
• Italian
• Portuguese
• Dutch
• Swedish
• Norwegian
• Danish

Custom Stop Words

You can add custom stop words to exclude specific terms from being indexed:

Custom stop words are applied in addition to the default language-specific stop words. They are case-insensitive and apply across all languages.

Geo-Spatial Search

YetiSearch supports location-based searching using SQLite's R-tree spatial indexing:

Geo Utilities:

Indexing with Bounds:

Search Result Deduplication

By default, YetiSearch deduplicates results to show only the best matching chunk per document:

Highlighting

Search results can include highlighted matches:

Fuzzy Search

Enable fuzzy matching for typo tolerance:

Advanced Fuzzy Search Algorithms

YetiSearch supports multiple fuzzy matching algorithms for different use cases:

Available Fuzzy Algorithms:

1. Trigram (Default) - Best overall accuracy and performance

   • Breaks words into 3-character sequences for matching
   • Excellent for most use cases
   • Good balance of speed and accuracy

2. Jaro-Winkler - Optimized for short strings

   • Great for names, titles, and short text
   • Favors matches with common prefixes
   • Very fast performance
3. Levenshtein - Edit distance algorithm
   • Counts insertions, deletions, and substitutions
   • Most flexible but requires term indexing
   • Best for handling complex typos

Configuration Options:

• fuzzy_algorithm: Choose between 'trigram' (default), 'jaro_winkler', or 'levenshtein'
• levenshtein_threshold: Maximum edit distance allowed for Levenshtein (1-3 recommended)
  • 1 = Single character changes only (fastest)
  • 2 = Up to 2 character edits (balanced)
  • 3 = Up to 3 character edits (most flexible but slower)
• min_term_frequency: Minimum occurrences for a term to be considered for fuzzy matching
• max_indexed_terms: Maximum number of indexed terms to check (affects performance)
• max_fuzzy_variations: Maximum fuzzy variations generated per search term
• fuzzy_score_penalty: Score reduction factor for fuzzy matches (0.0 = no penalty, 1.0 = zero score)
• indexed_terms_cache_ttl: How long to cache the indexed terms list (seconds)

Performance Considerations:

Different algorithms have different performance characteristics:

• Trigram: Fast indexing and searching, no additional term indexing required
• Jaro-Winkler: Very fast, ideal for short text matching
• Levenshtein: Requires term indexing, impacting indexing performance (~295 docs/sec vs ~670 docs/sec)

Term indexing is only performed when fuzzy_algorithm is set to 'levenshtein'. For most use cases, 'trigram' provides the best balance of accuracy and performance.

Query Result Caching

YetiSearch includes built-in query result caching to dramatically improve performance for repeated searches:

Cache Features:

• Automatic invalidation - Cache clears when documents are added, updated, or deleted
• LRU eviction - Least recently used entries are removed when cache is full
• SQLite-based storage - Cache persists across PHP requests
• Hit tracking - Monitor cache effectiveness with built-in statistics

Cache Management:

Performance Impact:

• First query: 5-30ms (depending on complexity)
• Cached query: 0.1-0.5ms (10-100x faster)
• Minimal memory overhead (cache stored in SQLite)
• No impact on indexing performance

Best Practices:

• Enable caching for production environments
• Set TTL based on your content update frequency
• Monitor hit rates to optimize cache size
• Use cache warming for predictable query patterns

Multi-Column FTS and Field Weighting

YetiSearch now supports multi-column FTS indexing for superior field weighting and performance:

Benefits of Multi-Column FTS:

• Native BM25 field weighting - SQLite applies weights directly during search
• ~5% faster performance - 6.76ms vs 7.09ms average query time
• Better relevance - Documents with matches in high-weight fields rank much higher
• Exact match detection - Perfect field matches get 100+ point boost

Two-Pass Search Strategy

For maximum precision, enable the optional two-pass search:

When to Use Two-Pass Search:

• When title/heading matches are critical
• For navigation/documentation searches
• When users expect exact title matches first
• Trade-off: ~2.3x slower but more precise

Migration Guide for v2.1.0

Upgrading Existing Indexes

Existing indexes continue to work but won't benefit from multi-column FTS. To upgrade:

Configuration Changes

Update your configuration to use new defaults:

Performance Comparison

Based on A/B testing with real-world data:

Recommendation: Use the default multi-column FTS for best balance of performance and relevance. Enable two-pass search only when title/heading matches are critical.

Real-World Example: Documentation Search

Here's how the v2.1.0 improvements solve the "scheduler" ranking problem:

Key Improvements Demonstrated:

• Exact title match "Scheduler" now scores 150+ (vs ~22 before)
• Field weights are properly applied through native BM25
• Exact match boost ensures correct spelling ranks higher
• Fuzzy search preserves exact match priority

Performance Optimization Tips:

Algorithm Benchmarking:

YetiSearch includes built-in benchmarking tools to help you choose the best fuzzy algorithm for your use case:

Faceted Search

Get aggregated counts for categories, tags, etc:

Architecture

See the architecture overview diagram and component notes in docs/architecture-overview.md.

Geo Search

YetiSearch supports location filtering and sorting with SQLite R-tree and accurate distances:

• Accurate distances: uses Haversine great‑circle distance (meters) when SQLite math functions are available; otherwise falls back to a planar approximation.
• near radius filter: radius (in meters) is applied in SQL using the computed distance for better performance and correctness.
• within bounds: supports standard bounding boxes; if the bounds cross the antimeridian (west > east), the query splits into two longitude ranges.
• Distance sorting: include a sort‑by‑distance option in your query for nearest‑first results.

Units

• Radius units default to meters. You can switch per‑query via units: 'km' | 'mi' inside the geo filter, or set a global default in search.geo_units (e.g., 'km').

Example (PHP):

Note

• R-tree is used when available; if not, geo search gracefully degrades but may be slower. Ensure your SQLite build has RTREE (check with scripts/check_sqlite_features.php).

Global Units & Composite Scoring

• Default units (global):

• Blend text relevance with distance using an exponential decay. Useful for “closest best” ranking:

Guidance

• Start with distance_weight between 0.3–0.6; increase if proximity should dominate when text scores are similar.
• Tune distance_decay_k by typical search radius; e.g., 0.005–0.02 for city‑scale queries.

Distance Facets

Bucket results by distance from a point to power UI filters.

k‑Nearest Neighbors (k‑NN)

Return the k nearest documents by distance, optionally clamped by max distance:

YetiSearch follows a modular architecture with clear separation of concerns:

Key Components

• Analyzer: Tokenizes and processes text (stemming, stop words, etc.)
• Indexer: Manages document indexing and updates
• SearchEngine: Handles search queries and result processing
• Storage: Abstracts the storage backend (currently SQLite)

Testing

YetiSearch includes comprehensive test coverage. Run tests using various commands:

Basic Testing

Coverage Reports

Filtered Testing

Advanced Testing

Static Analysis

API Reference

YetiSearch Class

Document Structure

Documents are represented as associative arrays with the following structure:

Content vs Metadata

Understanding the distinction between content and metadata fields:

Content Fields:

• Are indexed and searchable - these fields are analyzed, tokenized, and can be found via search queries
• Affect relevance scoring - matches in content fields contribute to the document's search score
• Support field boosting - you can make certain fields more important for ranking
• Are returned in search results by default
• Examples: title, body, description, tags, author, category

Metadata Fields:

• Are NOT indexed - stored in the database but not searchable
• Don't affect search scoring - won't influence relevance ranking
• Are returned in results - currently included but could be made optional
• Useful for filtering - can still filter results by metadata values using filters
• Examples: prices, stock counts, internal IDs, timestamps, flags, view counts

When to use metadata:

This separation improves performance (less data to index), prevents false matches (searching "42" won't find products with 42 in stock), and keeps your search index focused on actual searchable content.

SearchQuery Model

Result Structure

Search results are returned as an associative array:

Performance Tips

1. Index Configuration

   • Use appropriate field boosts - don't over-boost
   • Only index fields you need to search
   • Use metadata for non-searchable data
   • Configure reasonable chunk sizes (default 1000 chars works well)

2. Search Optimization

   • Use field-specific searches when possible: inFields(['title'])
   • Enable unique_by_route (default) to avoid duplicate documents
   • Use filters instead of text queries for exact matches
   • Limit results with reasonable page sizes
3. Storage Optimization
   • Run optimize() periodically on large indexes
   • Use WAL mode for better concurrency (default)
   • Consider separate indexes for different content types

Error Handling

Performance

YetiSearch is designed for high performance with minimal resource usage. Here are real-world benchmarks and performance characteristics.

Benchmark Results

Tested on M4 MacBook Pro with PHP 8.3, using a dataset of 32,000 movies:

Indexing Performance

Search Performance

Real-World Example

From the movie database benchmark:

• Dataset: 32k movies with title, overview, genres
• Index Size: ~200MB on disk
• Indexing Time: 7.27 seconds (~4,420 movies/sec)
• Search Examples:
  • "Harry Potter" (exact) → results in 4.7ms
  • "Matrix" (exact) -> results in 0.47ms
  • "Lilo and Stich" (fuzzy) → "Lilo & Stitch" in 26ms
  • "Cristopher Nolan" (fuzzy) → "Christopher Nolan" films in 32ms

Performance Characteristics

1. Linear Scalability

• Performance scales linearly with document count
• 100k documents ≈ 10x the time of 10k documents
• No exponential performance degradation

2. Memory Efficiency

• SQLite backend provides excellent memory management
• Only active data kept in memory
• Configurable cache sizes for different workloads

3. Disk I/O Optimization

• Write-Ahead Logging (WAL) for concurrent access
• Batch operations reduce disk writes
• Automatic index optimization

Performance Tuning

For Maximum Indexing Speed

For Fastest Searches

For Best Accuracy

Bottlenecks and Solutions

Comparison with Other Solutions

Best Practices for Performance

1. Index Design

   • Create separate indexes for different content types
   • Use appropriate field boosts
   • Only index searchable content

2. Query Optimization

   • Use field-specific searches when possible
   • Limit results appropriately
   • Enable result caching for repeated queries

3. Maintenance

   • Run optimize() during low-traffic periods
   • Monitor index size and split if needed
   • Clear old cache entries periodically
4. Hardware Considerations
   • SSD storage recommended for large indexes
   • More RAM allows larger caches
   • Multi-core CPUs benefit batch operations

Type-Ahead Setup

For as-you-type search, enable fuzzy matching and (optionally) last-token prefixing. Debounce input by 200–300ms on the client.

CLI Demo

Try an interactive demonstration that seeds a small dataset, prints suggestions, and shows as‑you‑type results:

Or run a single query:

Notes

• Interactive mode updates results after each character (min length 2). On macOS/Linux, it uses raw TTY mode; on unsupported environments it falls back to line input.

Weighted FTS and Prefix (Optional)

You can enable multi-column FTS5 and weighted BM25 to boost important fields (e.g., title, tags). Prefix indexing improves strict prefix matches for type-ahead.

Migration helper:

Suggestions

Use suggest(index, term, options) to power a dropdown for type‑ahead. Suggestions are ranked by frequency across fuzzy variants and boosted when the title contains or starts with the variant.

Tips

• Pair with ['fuzzy'=>true,'fuzzy_last_token_only'=>true,'prefix_last_token'=>true] on search for a smooth type‑ahead experience.
• For short terms (names/titles), try fuzzy_algorithm='jaro_winkler'; for general text, use trigram.

Synonyms

Enable query‑time synonyms expansion to improve recall for known aliases and abbreviations.

Config (array or JSON file):

Behavior

• Expands tokens before building the FTS query. Multi‑word synonyms are added as quoted phrases.
• Exact phrase is built from the original tokens; synonyms are ORed in. Fuzzy still works independently.
• Use a small, targeted list to avoid noise; adjust synonyms_max_expansions if needed.

DSL (Domain Specific Language)

YetiSearch now supports a powerful DSL for building complex queries with multiple syntaxes. For comprehensive documentation with migration guide and advanced examples, see docs/DSL.md.

Natural Language Query Syntax

Write queries using SQL-like syntax:

JSON API-Compliant URL Parameters

Support for standard REST API query patterns:

Example URL: ?filter[category][eq]=tech&filter[tags][in]=go,php&sort=-date&page[limit]=10

Fluent PHP Interface

Build queries programmatically:

DSL Features

• Operators: =, !=, >, <, >=, <=, LIKE, IN, NOT IN
• Logical: AND, OR, grouped conditions with parentheses
• Keywords: FIELDS, SORT, PAGE, LIMIT, OFFSET, FUZZY, NEAR, WITHIN
• Geo Queries: Support for location-based filtering and sorting
• Field Aliases: Map user-friendly names to actual field names
• Metadata Fields: Automatic handling of filterable/sortable attributes
• Negation: Use - prefix to negate conditions

Metadata Fields

YetiSearch distinguishes between content (searchable text) and metadata (filterable attributes):

Common metadata fields like author, status, price, views, etc. are automatically recognized. See docs/DSL.md for complete documentation.

CLI

A simple CLI is included for quick testing of search, suggestions, geo nearest, and distance facets.

Install deps if you haven't:

Examples

• Search (as‑you‑type style):

• DSL Search:

• URL Parameter Search:

• Suggestions:

• k‑NN nearest 5 around NYC (km):

• Distance facets (<= 1/3/5 km):

Synonyms example

Common flags

• --db=PATH (default benchmarks/benchmark.db)
• --synonyms=PATH (default examples/synonyms.json)
• --geo-units=m|km|mi (default meters)

Future Feature Ideas

The following features are ideas for future releases:

Index Management Enhancements

• Index Aliases - Create aliases for indexes to simplify management and allow seamless index switching
• Index Templates - Define templates for consistent index configuration across similar content types
• Automatic Index Routing - Route documents to appropriate indexes based on document properties
• Real-time Index Synchronization - Synchronize data between multiple indexes in real-time
• Index Versioning and Migrations - Support for index schema evolution with migration tools

Language and Analysis

• Automatic Language Detection - Detect document language automatically instead of defaulting to English
• Custom Analyzer Plugins - Allow custom text analysis plugins for specialized content
• Phonetic Matching - Support for soundex/metaphone matching for name searches
• Synonym Support - Configure synonyms for enhanced search matching

Search Enhancements

• Query DSL - Advanced query language for complex search expressions
• Search Templates - Save and reuse common search patterns
• More Like This - Find similar documents based on content similarity
• Search Analytics - Built-in analytics for search queries and results
• Full Content Result - Option to return full document content in search results

Performance and Scalability

• Distributed Search - Support for searching across multiple YetiSearch instances
• Index Sharding - Split large indexes across multiple shards
• Query Caching Improvements - More sophisticated caching strategies
• Bulk Operations API - Optimized bulk indexing and updates

Integration Features

• Webhook Support - Notify external systems of index changes
• Import/Export Tools - Tools for data migration between different search systems
• REST API - HTTP API for remote access to YetiSearch functionality
• GraphQL Support - GraphQL endpoint for flexible data querying

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Run tests (composer test:verbose)
4. Commit your changes (git commit -m 'Add amazing feature')
5. Push to the branch (git push origin feature/amazing-feature)
6. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Credits

YetiSearch is maintained by the YetiSearch Team and contributors.

Special thanks to:

• The SQLite team for the excellent FTS5 extension
• The PHP community for continuous inspiration
• All contributors who help make YetiSearch better