Libraries tagged by tokenizer

TokenizerX calculates required tokens for given prompt

Go to Download

Tokenizer

Go to Download

Native byte-level BPE (tiktoken-compatible: cl100k_base/o200k_base) plus WordPiece and Unigram tokenizers for PHP 8.3+, with a process-shared vocab cache. Loads HuggingFace tokenizer.json models and counts Claude/Gemini tokens via their APIs.

Go to Download

支持 scws, phpanalysis 和 jieba 的中文分词器

Go to Download

Go to Download

Provider-agnostic LLM token counter for PHP 8.3+. Supports Tiktoken BPE (OpenAI, Claude approx.), HuggingFace JSON vocabularies (DeepSeek, LLaMA, Mistral), SentencePiece (Gemini), and extensible custom strategies.

Go to Download

Thruster Tokenizer Component

Go to Download

The tokenizer component for super-kernel.

Go to Download

Library for creating an universal tokenizer

Go to Download

A generic Tokenizer

Go to Download

Create and validate token for user data

Go to Download

Tokenizer

Go to Download

Easy uri tokens creator and validator

Go to Download

Simple wrapper class around PHP's built in tokenize functions'

Go to Download

Osirisgate token management.

Go to Download