Libraries tagged by tokenizer

Fast and extensible PHP code analysis and transformation

Go to Download

Fast, high-accuracy Thai word segmentation & typographic line breaker for PHP and Laravel (dompdf, mPDF, Web)

Go to Download

A lightweight PHP utility to extract the Fully Qualified Class Name (FQCN) from a PHP source file using the built-in tokenizer.

Go to Download

Attribute-driven PHP lexer and parser toolkit with enum tokens, grammar recipes, diagnostics, and AST/BNF debugging.

Go to Download

PHP version of tiktoken. PHP 7.4+ required.

Go to Download

Tokenizer-agnostic text preprocessor for trimming and optimising content for LLM context windows.

Go to Download

Small utilities that assist in developing applications

Go to Download

A very lightweight PHP class that reformats unreadable and computer-generated SQL query statements to human-friendly, readable text.

Go to Download

PHP BPE Text Encoder/Decoder for GPT-2 / GPT-3. Based on coderevolutionplugins/gpt-3-encoder-php

Go to Download

Manipulation of php code, automated refactoring based on phps tokenizer

Go to Download

Magento translation tool that utilizes PHP Tokenizer in order to generate csv files from PHP, XML, PHTML code

Go to Download

PHP SDK for Service Framework authentication and token generation.

Go to Download

TokenizerX calculates required tokens for given prompt

Go to Download

Tokenizer

Go to Download

Native byte-level BPE (tiktoken-compatible: cl100k_base/o200k_base) plus WordPiece and Unigram tokenizers for PHP 8.3+, with a process-shared vocab cache. Loads HuggingFace tokenizer.json models and counts Claude/Gemini tokens via their APIs.

Go to Download