Libraries tagged by tokenize
mistralys/jtokenizer
4637 Downloads
Modernized, composer compatible version of Tim Whitlock's JTokenizer PHP Javascript parser and tokenizer library.
ride/lib-tokenizer
6915 Downloads
Tokenizer library of the Ride framework.
ricwein/tokenizer
971 Downloads
configurable string tokenizer
kerasai/behat-tokenizer
19861 Downloads
Token replacement for Behat test suite.
funivan/php-tokenizer
38440 Downloads
Wrapper around token_get_all. Easy to extract and modify php code.
fi1a/tokenizer
3216 Downloads
Лексический анализатор строк (tokenizer). Разбор входной последовательности символов на распознанные группы (лексемы).
yannickl88/css-tokenizer
30633 Downloads
CSS tokenizer
turanjanin/serbian-language-tools
1932 Downloads
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.
popsul/php-token-reflection
76953 Downloads
Library emulating the PHP internal reflection using just the tokenized source code.
hybridlogic/classifier
1435 Downloads
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.
hippophp/hippo
32 Downloads
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
guttedgarden/tiktoken
9992 Downloads
PHP 7.4 version of tiktoken
didix16/php-grammar
6379 Downloads
A simple library for make Lexer and Parsers to build a language
cxrxexo/tiktoken
2020 Downloads
PHP version of tiktoken. PHP 7.4+ required.
laurynasgadl/travers
455608 Downloads
Travers - PHP array traversing via tokenized paths