Download the PHP package yaroslavpopovic/laravel-text-chunker without Composer

Laravel Text Chunker

A Laravel package that provides flexible, strategy-based text chunking capabilities for AI/LLM applications. Split text into smaller segments using character count, token count, sentence boundaries, or markdown-aware strategies with a fluent, Laravel-friendly API.

Perfect for:

• Optimizing API calls to LLM providers like OpenAI by chunking text to fit token limits
• Implementing RAG (Retrieval-Augmented Generation) systems with context-aware chunks
• Preserving markdown structure when splitting documentation or content
• Creating custom text splitting logic for domain-specific needs

Requirements

• PHP 8.3 or higher
• Laravel 11.x or 12.x

Installation

Install the package via Composer:

The package will automatically register itself via Laravel's auto-discovery.

Configuration

Optionally, publish the configuration file:

This will create a config/text-chunker.php file where you can customize default settings:

Basic Usage

Character-Based Chunking

Split text at exact character count boundaries:

Token-Based Chunking

Split text by OpenAI token count (perfect for API optimization):

Supported Models:

• gpt-4
• gpt-3.5-turbo
• text-davinci-003
• And other models supported by the tiktoken library

Sentence-Based Chunking

Split text at sentence boundaries:

Markdown-Aware Chunking

Preserve markdown structure when chunking:

php function example() { return "code block"; }

Advanced Features

Overlap for Context Preservation

Add percentage-based overlap between chunks to maintain context (ideal for RAG systems):

Overlap works with all strategies:

• Character strategy: 20% of characters overlap
• Token strategy: 20% of tokens overlap
• Sentence strategy: 20% of sentences overlap (rounded)
• Markdown strategy: 20% overlap while preserving element boundaries

Chunk Value Objects

Each chunk is returned as an immutable value object with metadata:

Using the Manager Directly

Instead of the facade, you can inject the manager:

Custom Strategies

Create your own chunking strategies by implementing the ChunkerStrategyInterface:

Step 1: Create Strategy Class

Step 2: Register Strategy

Option A: Via Configuration

Add to config/text-chunker.php:

Option B: At Runtime

Option C: In a Service Provider

Fluent API Reference

The package provides a fluent, chainable API:

Method Details:

• strategy(string $name, array $options = []): Select chunking strategy

  • Built-in strategies: 'character', 'token', 'sentence', 'markdown'
  • Options vary by strategy (e.g., ['model' => 'gpt-4'] for token strategy)

• size(int $size): Set target chunk size (required)

  • Interpretation depends on strategy (characters, tokens, sentences)
  • Must be greater than zero

• overlap(int $percentage): Set overlap between chunks (optional)

  • Percentage: 0-100
  • Copies content from end of previous chunk to start of next chunk
• chunk(string $text): Execute chunking and return array of Chunk objects
  • Validates all parameters (deferred validation)
  • Throws ChunkerException on validation failures
  • Returns array<int, Chunk>

Validation and Error Handling

All validation is deferred until the chunk() method is called:

Common Exceptions:

• Size not set: "Chunk size must be set before calling chunk()"
• Size <= 0: "Chunk size must be greater than zero"
• Invalid overlap: "Overlap percentage must be between 0 and 100"
• Empty text: "Text cannot be empty"
• Unknown strategy: "Unknown chunking strategy: xyz. Available strategies: character, token, sentence, markdown"
• Invalid token model: "Unsupported model: xyz"

Testing

Run with coverage:

Code Quality

Format code with Laravel Pint:

Run static analysis with PHPStan:

Changelog

Please see CHANGELOG for more information on what has changed recently.

Contributing

Please see CONTRIBUTING for details.

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities.

Credits

• Travis Tomka
• All Contributors

License

The MIT License (MIT). Please see License File for more information.