Libraries tagged by tokenizer

Framework for building BNF LR(1) parsers

Go to Download

Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.

Go to Download

Word breaker is Thai wording separator for using with fulltext software.

Go to Download

A polyfill for PHP 8.0's PhpToken class

Go to Download

Fast and extensible PHP code analysis and transformation

Go to Download

A lightweight PHP utility to extract the Fully Qualified Class Name (FQCN) from a PHP source file using the built-in tokenizer.

Go to Download

Parse search queries into structured data with field searches, boolean logic, ranges, lists, and adapters.

Go to Download

Attribute-driven PHP lexer and parser toolkit with enum tokens, grammar recipes, diagnostics, and AST/BNF debugging.

Go to Download

Lazy class discovery with composable filters and listener notification

Go to Download

Small utilities that assist in developing applications

Go to Download

A high-performance, semantic syntax highlighter for PHP 8.4+ with zero dependencies. Features context-aware parsing, language embedding, and compatibility with Highlight.js and Prism themes.

Go to Download

Manipulation of php code, automated refactoring based on phps tokenizer

Go to Download

Magento translation tool that utilizes PHP Tokenizer in order to generate csv files from PHP, XML, PHTML code

Go to Download

PHP SDK for Service Framework authentication and token generation.

Go to Download

PHP Library to tokenize various inputs.

Go to Download