Libraries tagged by tokenize

PHP Library to tokenize various inputs.

Go to Download

TokenizerX calculates required tokens for given prompt

Go to Download

Tokenizer

Go to Download

An experiment with using the PHP tokenizer functions.

Go to Download

Native byte-level BPE (tiktoken-compatible: cl100k_base/o200k_base) plus WordPiece and Unigram tokenizers for PHP 8.3+, with a process-shared vocab cache. Loads HuggingFace tokenizer.json models and counts Claude/Gemini tokens via their APIs.

Go to Download

支持 scws, phpanalysis 和 jieba 的中文分词器

Go to Download

Go to Download

Go to Download

Provider-agnostic LLM token counter for PHP 8.3+. Supports Tiktoken BPE (OpenAI, Claude approx.), HuggingFace JSON vocabularies (DeepSeek, LLaMA, Mistral), SentencePiece (Gemini), and extensible custom strategies.

Go to Download

Thruster Tokenizer Component

Go to Download

The tokenizer component for super-kernel.

Go to Download

Library for creating an universal tokenizer

Go to Download

A generic Tokenizer

Go to Download

A library to split a string into smaller tokens. This library is part of the SoloProyectos PHP API.

Go to Download

Create and validate token for user data

Go to Download