Libraries tagged by tokenizer
droath/laravel-text-chunker
2154 Downloads
Flexible, strategy-based text chunking for Laravel with support for character, token, sentence, and markdown-aware splitting. Perfect for AI/LLM applications and RAG systems.
herrera-io/annotations
135213 Downloads
A tokenizer for Doctrine annotations.
phootwork/tokenizer
218079 Downloads
Tokenizer for PHP code
neuron-core/tokenizer
1334 Downloads
High-Performance Tokenizer implementation in PHP
mathematicator-core/tokenizer
5297 Downloads
Math Tokenizer
fi1a/tokenizer
17696 Downloads
Лексический анализатор строк (tokenizer). Разбор входной последовательности символов на распознанные группы (лексемы).
componenta/tokenizer
1840 Downloads
Token-based PHP source inspection helpers for Componenta
artemeon/tokenizer
94783 Downloads
Tokenizer for php
semji/gpt-3-tokenizer-php
159837 Downloads
PHP Text Tokenizer for GPT models
infinityloop-dev/graphpinator-tokenizer
44825 Downloads
Tokenizer subproject for GraPHPinator: GraphQL server implementation for PHP.
aisk/mistral-tokenizer
22741 Downloads
PHP implementation of Mistral's tokenizers
yurunsoft/tiktoken
2661 Downloads
PHP version of tiktoken
phplrt/lexer-builder
5197 Downloads
The phplrt component for assembling a lexer
mosuka/litsea
11 Downloads
PHP extension for Litsea, a compact word segmentation and POS tagging library (no embedded models)
lindera/lindera
3 Downloads
PHP extension for Lindera, a multilingual morphological analysis library (no embedded dictionaries)