Libraries tagged by tokenizer
tacoberu/bnf
2376 Downloads
Framework for building BNF like parsers.
popsul/php-token-reflection
76952 Downloads
Library emulating the PHP internal reflection using just the tokenized source code.
hybridlogic/classifier
1435 Downloads
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.
hippophp/hippo
32 Downloads
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
guttedgarden/tiktoken
9992 Downloads
PHP 7.4 version of tiktoken
didix16/php-grammar
6379 Downloads
A simple library for make Lexer and Parsers to build a language
cxrxexo/tiktoken
2020 Downloads
PHP version of tiktoken. PHP 7.4+ required.
ze/tokenizer-gpt3
1412 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
tanuel/tokenizer
14 Downloads
A tokenizer for PHP
robinthehood/tokenizer
816 Downloads
With Tokenizer you can easily convert a string to a stream or array of tokens.
mathematicator-core/tokenizer
4414 Downloads
Math Tokenizer
hippophp/tokenizer
1101 Downloads
Wrapper around token_get_all
everstu/gpt3-tokenizer
74 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
sastrawi/tokenizer
946 Downloads
PHP library that allows you to tokenize Bahasa Indonesia.
fiv/tokenizer
248 Downloads
Extreme and flexible library for parsing and modify php code