Libraries tagged by tokenized
vanry/tokenizer
2 Downloads
支持 scws, phpanalysis 和 jieba 的中文分词器
ueef/tokenizer
84 Downloads
ueef/auth-tokenizer
901 Downloads
token27/nexus-ai-tokenizer
25 Downloads
Provider-agnostic LLM token counter for PHP 8.3+. Supports Tiktoken BPE (OpenAI, Claude approx.), HuggingFace JSON vocabularies (DeepSeek, LLaMA, Mistral), SentencePiece (Gemini), and extensible custom strategies.
thruster/tokenizer
6 Downloads
Thruster Tokenizer Component
super-kernel/tokenizer
45 Downloads
The tokenizer component for super-kernel.
sterzik/tokenizer
1 Downloads
Library for creating an universal tokenizer
stella-maris/tokenizer
1 Downloads
A generic Tokenizer
soloproyectos-php/text-tokenizer
891 Downloads
A library to split a string into smaller tokens. This library is part of the SoloProyectos PHP API.
sincco/tokenizer
180 Downloads
Create and validate token for user data
scripturadesign/tokenizer
7 Downloads
Tokenizer
pierotto/tokenizer-bundle
19 Downloads
The package is used for secure data exchange between two parties using a JWT token.
picios/tokenizer
8 Downloads
Easy uri tokens creator and validator
phpfacile/string-tokenizer
41 Downloads
Library to parse a string (especially to extract text and keywords identified by a begin and end delimiter and to substitute keywords by values)
philwaters/tokenizer
12 Downloads
Simple wrapper class around PHP's built in tokenize functions'