Libraries tagged by token-counting

token27/nexus-ai-tokenizer

0 Favers
25 Downloads

Provider-agnostic LLM token counter for PHP 8.3+. Supports Tiktoken BPE (OpenAI, Claude approx.), HuggingFace JSON vocabularies (DeepSeek, LLaMA, Mistral), SentencePiece (Gemini), and extensible custom strategies.

Go to Download

arnaudleroy-studio/mohitkhare

0 Favers
0 Downloads

Developer utilities and token counting tools.

Go to Download