Libraries tagged by cl100k_base

rajentrivedi/tokenizer-x
92 Favers
253305 Downloads

TokenizerX calculates required tokens for given prompt

Go to Download

danny50610/bpe-tokeniser
10 Favers
458721 Downloads

PHP port for openai/tiktoken (most)

Go to Download

your1/tokenizer-x
0 Favers
273 Downloads

TokenizerX calculates required tokens for given prompt

Go to Download

juniora/tokenizer-x
0 Favers
1 Downloads

TokenizerX calculates required tokens for given prompt

Go to Download

webrek/tokenizers
0 Favers
0 Downloads

Native byte-level BPE (tiktoken-compatible: cl100k_base/o200k_base) plus WordPiece and Unigram tokenizers for PHP 8.3+, with a process-shared vocab cache. Loads HuggingFace tokenizer.json models and counts Claude/Gemini tokens via their APIs.

Go to Download