Libraries tagged by sentencepiece

codewithkyrian/tokenizers
20 Favers
3013 Downloads

Fast, pure-PHP tokenizer library compatible with Hugging Face tokenizers for encoding and decoding text

Go to Download

textualization/sentencepiece
1 Favers
1813 Downloads

Google SentencePiece bindings using FFI and a C adapter.

Go to Download

token27/nexus-ai-tokenizer
0 Favers
25 Downloads

Provider-agnostic LLM token counter for PHP 8.3+. Supports Tiktoken BPE (OpenAI, Claude approx.), HuggingFace JSON vocabularies (DeepSeek, LLaMA, Mistral), SentencePiece (Gemini), and extensible custom strategies.

Go to Download