Libraries tagged by tokenize
ltd-beget/sphinx-configuration-tokenizer
24114 Downloads
tokenize sphinx configuration and that's all, folks.
textualization/ropherta-tokenizer
1291 Downloads
GPT3Tokenizer (BPE) with Roberta-base vocabulary.
ride/lib-tokenizer
8133 Downloads
Tokenizer library of the Ride framework.
kerasai/behat-tokenizer
21287 Downloads
Token replacement for Behat test suite.
fi1a/tokenizer
17474 Downloads
Лексический анализатор строк (tokenizer). Разбор входной последовательности символов на распознанные группы (лексемы).
yannickl88/css-tokenizer
31012 Downloads
CSS tokenizer
hippophp/tokenizer
1379 Downloads
Wrapper around token_get_all
h4kuna/latte-php-tokenizer
26537 Downloads
Transform macro args to array.
turanjanin/serbian-language-tools
2364 Downloads
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.
tacoberu/bnf
3367 Downloads
Framework for building BNF like parsers.
shikiphp/shikiphp
399 Downloads
Pure PHP port of Shiki. Syntax-highlight code with VS Code themes and TextMate grammars — no extensions, no Node, no Oniguruma binding.
programmerhasan/bkash
686 Downloads
Bkash Payment Integration for Laravel.
popsul/php-token-reflection
78961 Downloads
Library emulating the PHP internal reflection using just the tokenized source code.
hybridlogic/classifier
1447 Downloads
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.
frolaxhq/laravel-llm-tokenkit
1754 Downloads
Token estimation, cost calculation & context-window utilities for LLM-powered Laravel apps