Libraries tagged by tokeniser
kerasai/behat-tokenizer
19817 Downloads
Token replacement for Behat test suite.
fi1a/tokenizer
3173 Downloads
Лексический анализатор строк (tokenizer). Разбор входной последовательности символов на распознанные группы (лексемы).
averay/tokenised-strings
208 Downloads
A lightweight system for rendering tokens in templated strings.
yannickl88/css-tokenizer
30596 Downloads
CSS tokenizer
turanjanin/serbian-language-tools
1921 Downloads
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.
popsul/php-token-reflection
76914 Downloads
Library emulating the PHP internal reflection using just the tokenized source code.
hybridlogic/classifier
1433 Downloads
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.
hippophp/hippo
32 Downloads
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
didix16/php-grammar
6334 Downloads
A simple library for make Lexer and Parsers to build a language
cxrxexo/tiktoken
1990 Downloads
PHP version of tiktoken. PHP 7.4+ required.
ze/tokenizer-gpt3
1405 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
textualization/ropherta-tokenizer
56 Downloads
GPT3Tokenizer (BPE) with Roberta-base vocabulary.
tanuel/tokenizer
14 Downloads
A tokenizer for PHP
ricwein/tokenizer
952 Downloads
configurable string tokenizer
mathematicator-core/tokenizer
4414 Downloads
Math Tokenizer