Libraries tagged by tokeniser

A high-performance, semantic syntax highlighter for PHP 8.4+ with zero dependencies. Features context-aware parsing, language embedding, and compatibility with Highlight.js and Prism themes.

Go to Download

A very lightweight PHP class that reformats unreadable and computer-generated SQL query statements to human-friendly, readable text.

Go to Download

Manipulation of php code, automated refactoring based on phps tokenizer

Go to Download

Laravel integration for the Paycorp (Bancstac) payment gateway. Supports Hosted Page flow, Real-Time tokenised payments, and Refund/Void.

Go to Download

Magento translation tool that utilizes PHP Tokenizer in order to generate csv files from PHP, XML, PHTML code

Go to Download

PHP SDK for Service Framework authentication and token generation.

Go to Download

PHP Library to tokenize various inputs.

Go to Download

TokenizerX calculates required tokens for given prompt

Go to Download

Tokenizer

Go to Download

An experiment with using the PHP tokenizer functions.

Go to Download

Native byte-level BPE (tiktoken-compatible: cl100k_base/o200k_base) plus WordPiece and Unigram tokenizers for PHP 8.3+, with a process-shared vocab cache. Loads HuggingFace tokenizer.json models and counts Claude/Gemini tokens via their APIs.

Go to Download

支持 scws, phpanalysis 和 jieba 的中文分词器

Go to Download

Go to Download

Go to Download

Provider-agnostic LLM token counter for PHP 8.3+. Supports Tiktoken BPE (OpenAI, Claude approx.), HuggingFace JSON vocabularies (DeepSeek, LLaMA, Mistral), SentencePiece (Gemini), and extensible custom strategies.

Go to Download