Libraries tagged by tokenize
ze/tokenizer-gpt3
1277 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
kerasai/behat-tokenizer
19671 Downloads
Token replacement for Behat test suite.
gollumsf/url-tokenizer-bundle
1446 Downloads
Simple url tokenizer
funivan/php-tokenizer
38433 Downloads
Wrapper around token_get_all. Easy to extract and modify php code.
fi1a/tokenizer
2995 Downloads
Лексический анализатор строк (tokenizer). Разбор входной последовательности символов на распознанные группы (лексемы).
yannickl88/css-tokenizer
30506 Downloads
CSS tokenizer
kevintweber/html-tokenizer
46157 Downloads
Will tokenize HTML.
h4kuna/latte-php-tokenizer
11945 Downloads
Transform macro args to array.
tacoberu/bnf
2253 Downloads
Framework for building BNF like parsers.
popsul/php-token-reflection
76848 Downloads
Library emulating the PHP internal reflection using just the tokenized source code.
hybridlogic/classifier
1431 Downloads
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.
hippophp/hippo
32 Downloads
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
didix16/php-grammar
6076 Downloads
A simple library for make Lexer and Parsers to build a language
tanuel/tokenizer
14 Downloads
A tokenizer for PHP
mathematicator-core/tokenizer
4412 Downloads
Math Tokenizer