Libraries tagged by tokenized
megao-phpunit/php-token-stream
45 Downloads
Wrapper around PHP's tokenizer extension.
legacyphpdoctor/yethee_tiktoken-php-7-4
1 Downloads
PHP version of tiktoken
jojo1981/json-path-ast-builder
4 Downloads
Json Path AST builder
jojo1981/json-ast-builder
6 Downloads
Json AST builder
jamesfrost/statistical-classifier
2680 Downloads
A PHP implementation of Complement Naive Bayes and SVM statistical classifiers, including a structure for building other classifier, multiple data sources and multiple caching backends
html5/htmlreader
3306 Downloads
Html5 stream tokenizer/reader (not using libxml)
hike/hike
5 Downloads
A consistent interface to tokenize, analyse and modify any text format. Programming languages, natural languages, markup languages
guanhui07/tiktoken
8 Downloads
PHP version of tiktoken
gorriecoe/silverstripe-sreg
137 Downloads
Simple tokenizer that allows you to use .ss template like variables in a dataobjects string.
fourkitchens/tiktoken
0 Downloads
PHP version of tiktoken
djiele/autoloader
11 Downloads
Another class loader using PHP tokenizer
cxrxexo/tiktoken
2135 Downloads
PHP version of tiktoken. PHP 7.4+ required.
codechap/yii3-context-trimmer
7 Downloads
Tokenizer-agnostic text preprocessor for trimming and optimising content for LLM context windows.
attla/token
3 Downloads
Turn everything into a unique encrypted JWT.
antto1/superclosure
161 Downloads
Serialize Closure objects, including their context and binding