Libraries tagged by tokenizer
octava/code-generator
1 Downloads
PHP code generator
nexxes/tokenmatcher
9 Downloads
PHP library to match tokenstreams (arrays of tokens) for later parsing.
muhfirdaus19/bahasata
6 Downloads
Text Processing PHP bahasa indonesia
megao-phpunit/php-token-stream
45 Downloads
Wrapper around PHP's tokenizer extension.
legacyphpdoctor/yethee_tiktoken-php-7-4
1 Downloads
PHP version of tiktoken
jamesfrost/statistical-classifier
2680 Downloads
A PHP implementation of Complement Naive Bayes and SVM statistical classifiers, including a structure for building other classifier, multiple data sources and multiple caching backends
html5/htmlreader
3306 Downloads
Html5 stream tokenizer/reader (not using libxml)
hike/hike
5 Downloads
A consistent interface to tokenize, analyse and modify any text format. Programming languages, natural languages, markup languages
guanhui07/tiktoken
8 Downloads
PHP version of tiktoken
gorriecoe/silverstripe-sreg
138 Downloads
Simple tokenizer that allows you to use .ss template like variables in a dataobjects string.
fourkitchens/tiktoken
0 Downloads
PHP version of tiktoken
djiele/autoloader
11 Downloads
Another class loader using PHP tokenizer
codechap/yii3-context-trimmer
8 Downloads
Tokenizer-agnostic text preprocessor for trimming and optimising content for LLM context windows.
attla/token
3 Downloads
Turn everything into a unique encrypted JWT.
apemsel/attributed-string
555 Downloads
A collection of fast and easy to use attributed string classes for PHP. Attributed strings can have multiple attributes for each character of the string and are for example used for word processors and natural language processing.