Download the PHP package codewithkyrian/tokenizers without Composer
On this page you can find all versions of the php package codewithkyrian/tokenizers. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Download codewithkyrian/tokenizers
More information about codewithkyrian/tokenizers
Files in codewithkyrian/tokenizers
Package tokenizers
Short Description Fast, pure-PHP tokenizer library compatible with Hugging Face tokenizers for encoding and decoding text
License MIT
FAQ
Example:
In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
- Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
- To use Composer is sometimes complicated. Especially for beginners.
- Composer needs much resources. Sometimes they are not available on a simple webspace.
- If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
- Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Informations about the package tokenizers
Tokenizers PHP
Tokenizers PHP is a lightweight, dependency-free PHP library for tokenizing text using the same tokenizers powering models on the Hugging Face Hub. Whether you're building LLM applications, search systems, or text processing pipelines, this library provides fast, accurate tokenization that matches the original model implementations.
Highlights
- Pure PHP — No FFI, no external binaries, no compiled extensions. Works everywhere PHP runs.
- Hub Integration — Load tokenizers from Hugging Face Hub with smart caching and manifest-based file checks.
- Flexible Loading — Load from local files, config arrays, or build custom tokenizers with the builder API.
- Fully Tested — Validated against BERT, GPT-2, Llama, Gemma, Qwen, RoBERTa, ALBERT, and more.
- Modern PHP — Built for PHP 8.2+ with strict types, readonly classes, and clean interfaces.
Installation
Install via Composer:
HTTP Client (for Hub loading)
Loading tokenizers from the Hugging Face Hub requires an HTTP client. We recommend Guzzle:
Quick Start
Loading Tokenizers
Tokenizers PHP provides multiple ways to load tokenizers depending on your use case.
From Hugging Face Hub
Load any tokenizer from the Hugging Face Hub by providing the model ID:
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
modelId |
string |
— | The model identifier on Hugging Face Hub (e.g., bert-base-uncased or org/model-name) |
cacheDir |
?string |
null |
Custom directory for caching downloaded files. Defaults to system cache directory |
revision |
?string |
'main' |
Specific version to load—can be a branch name, tag, or commit hash |
token |
?string |
null |
Hugging Face authentication token for accessing private or gated models |
Cache Directory Resolution
When cacheDir is not specified, the library automatically resolves the cache location:
- HF_HUB_CACHE — if set, used directly
- HF_HOME — if set,
$HF_HOME/hub - macOS —
~/Library/Caches/huggingface/hub - Linux —
$XDG_CACHE_HOME/huggingface/hubor~/.cache/huggingface/hub - Windows —
%LOCALAPPDATA%\huggingface\hub
Pass cacheDir to use a custom directory.
From Local Files
Load tokenizers from local JSON files:
This is useful when you've downloaded model files manually or are working in an offline environment.
From Configuration Array
Build a tokenizer from a raw configuration array:
Universal Loader
The load() method provides a convenient unified interface:
Accessing Configuration
The tokenizer stores its configuration and provides access via getConfig():
Common configuration keys:
model_max_length— Maximum sequence lengthremove_space— Whether to remove leading/trailing spacesdo_lowercase_and_remove_accent— Whether to lowercase and strip accentsclean_up_tokenization_spaces— Whether to clean up spaces during decoding
Encoding Text
The encode() method tokenizes text and returns an Encoding object containing the token IDs, tokens, and type IDs.
The Encoding Object
Encoding Options
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
text |
string |
— | The primary text to tokenize |
textPair |
?string |
null |
Optional second text for sequence pair tasks (e.g., question-answering) |
addSpecialTokens |
bool |
true |
Whether to add model-specific special tokens (like [CLS], [SEP]) |
Sentence Pairs
For tasks involving two text sequences (like question-answering or natural language inference), pass both texts:
The typeIds distinguish between the first sequence (0) and the second sequence (1), which many models use during attention computation.
Decoding Tokens
Convert token IDs back to human-readable text:
Decoding Options
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
ids |
int[] |
— | Array of token IDs to decode |
skipSpecialTokens |
bool |
true |
Whether to exclude special tokens from the output |
cleanup |
?bool |
null |
Whether to clean up tokenization artifacts (extra spaces). Uses model's config when null |
Cleanup Behavior
The cleanup parameter controls whether tokenization artifacts are cleaned:
When cleanup is null, the library respects the clean_up_tokenization_spaces setting from the model's configuration.
Custom Tokenizers with the Builder
For advanced use cases, build tokenizers from scratch using the fluent builder API:
Builder Methods
| Method | Description |
|---|---|
withModel(ModelInterface $model) |
Required. Set the tokenization model (BPE, WordPiece, Unigram) |
withNormalizer(NormalizerInterface $normalizer) |
Set text normalizer. Defaults to PassThroughNormalizer |
withPreTokenizer(PreTokenizerInterface $preTokenizer) |
Set pre-tokenizer. Defaults to IdentityPreTokenizer |
withPostProcessor(PostProcessorInterface $postProcessor) |
Set post-processor. Defaults to DefaultPostProcessor |
withDecoder(DecoderInterface $decoder) |
Set decoder. Defaults to FuseDecoder |
withAddedTokens(array $tokens) |
Add extra tokens to the vocabulary |
withSpecialTokens(array $tokens) |
Define special tokens (skipped during decode by default) |
withConfig(string $key, mixed $value) |
Set a configuration value (see common keys below) |
build() |
Build and return the Tokenizer instance |
Common config keys for withConfig():
'model_max_length'— Maximum sequence length'remove_space'— Remove leading/trailing spaces before normalization'do_lowercase_and_remove_accent'— Lowercase and strip accents'clean_up_tokenization_spaces'— Clean up spaces during decoding
The Tokenization Pipeline
Understanding the tokenization pipeline helps when debugging or customizing behavior. Each input text passes through these stages:
Components Reference
Normalizers
Normalizers clean and standardize input text before tokenization.
| Normalizer | Description |
|---|---|
BertNormalizer |
BERT-style: clean text, handle Chinese chars, lowercase, strip accents |
LowercaseNormalizer |
Convert all characters to lowercase |
NFCNormalizer |
Unicode NFC normalization |
NFKCNormalizer |
Unicode NFKC normalization |
NFKDNormalizer |
Unicode NFKD normalization |
StripNormalizer |
Strip leading/trailing whitespace |
StripAccentsNormalizer |
Remove accent marks from characters |
ReplaceNormalizer |
Replace patterns or strings |
PrependNormalizer |
Prepend a string to the input |
PrecompiledNormalizer |
Use precompiled normalization rules (for SentencePiece models) |
NormalizerSequence |
Chain multiple normalizers together |
PassThroughNormalizer |
No-op, passes text through unchanged |
Pre-tokenizers
Pre-tokenizers split text into smaller chunks before subword tokenization.
| Pre-tokenizer | Description |
|---|---|
BertPreTokenizer |
Split on whitespace and punctuation (BERT-style) |
ByteLevelPreTokenizer |
Convert to byte-level representation (GPT-2 style) |
WhitespacePreTokenizer |
Split on whitespace characters |
WhitespaceSplit |
Split only on whitespace, keep punctuation attached |
MetaspacePreTokenizer |
Replace spaces with ▁ (SentencePiece style) |
PunctuationPreTokenizer |
Split on punctuation characters |
DigitsPreTokenizer |
Isolate digit sequences |
SplitPreTokenizer |
Split using custom regex patterns |
PreTokenizerSequence |
Chain multiple pre-tokenizers together |
IdentityPreTokenizer |
No-op, returns text unchanged |
Models
Models perform the core subword tokenization algorithm.
| Model | Description |
|---|---|
BPEModel |
Byte-Pair Encoding - iteratively merges most frequent pairs |
WordPieceModel |
Greedy longest-match-first subword tokenization (BERT) |
UnigramModel |
Probabilistic subword selection (SentencePiece) |
FallbackModel |
Simple vocabulary lookup with unknown token fallback |
Post-processors
Post-processors add special tokens and structure to the tokenized output.
| Post-processor | Description |
|---|---|
BertPostProcessor |
Add [CLS] and [SEP] tokens |
RobertaPostProcessor |
Add <s> and </s> tokens with spacing |
TemplatePostProcessor |
Flexible template-based token insertion |
ByteLevelPostProcessor |
Handle byte-level special tokens |
PostProcessorSequence |
Chain multiple post-processors |
DefaultPostProcessor |
Minimal processing, no tokens added |
Decoders
Decoders convert tokens back to readable text.
| Decoder | Description |
|---|---|
ByteLevelDecoder |
Decode byte-level tokens back to UTF-8 |
WordPieceDecoder |
Handle ## continuation prefixes |
MetaspaceDecoder |
Convert ▁ back to spaces |
BPEDecoder |
Handle BPE-specific suffixes and spaces |
CTCDecoder |
Decode CTC (Connectionist Temporal Classification) output |
FuseDecoder |
Simply join tokens with optional separator |
ReplaceDecoder |
Replace specific patterns during decode |
StripDecoder |
Strip specific characters |
ByteFallbackDecoder |
Handle byte fallback tokens (e.g., <0x00>) |
DecoderSequence |
Chain multiple decoders together |
Extending the Library
All components implement simple interfaces that you can extend:
Available interfaces:
NormalizerInterface— Text normalizationPreTokenizerInterface— Pre-tokenization splittingModelInterface— Core tokenization algorithmPostProcessorInterface— Post-processing and special tokensDecoderInterface— Token-to-text conversion
Contributing
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Credits
- Kyrian Obikwelu — Creator and maintainer
- Hugging Face — Tokenizers specification and model hosting
- All contributors
Made with ❤️ for the PHP community
