Libraries tagged by tokenize
scheldejonas/phpunit-php-token-stream
205 Downloads
Wrapper around PHP's tokenizer extension.
octava/code-generator
1 Downloads
PHP code generator
nexxes/tokenmatcher
9 Downloads
PHP library to match tokenstreams (arrays of tokens) for later parsing.
muhfirdaus19/bahasata
6 Downloads
Text Processing PHP bahasa indonesia
megao-phpunit/php-token-stream
37 Downloads
Wrapper around PHP's tokenizer extension.
jamesfrost/statistical-classifier
2676 Downloads
A PHP implementation of Complement Naive Bayes and SVM statistical classifiers, including a structure for building other classifier, multiple data sources and multiple caching backends
html5/htmlreader
3243 Downloads
Html5 stream tokenizer/reader (not using libxml)
guanhui07/tiktoken
1 Downloads
PHP version of tiktoken
gorriecoe/silverstripe-sreg
132 Downloads
Simple tokenizer that allows you to use .ss template like variables in a dataobjects string.
ghostwriter/syntax
0 Downloads
Lexical Analysis library for PHP
fourkitchens/tiktoken
0 Downloads
PHP version of tiktoken
djiele/autoloader
7 Downloads
Another class loader using PHP tokenizer
crazzy501/gpt-3-encoder-php
274 Downloads
PHP BPE Text Encoder/Decoder for GPT-2 / GPT-3. Based on coderevolutionplugins/gpt-3-encoder-php
attla/data-token
16 Downloads
Turn everything into a token: randomized or predictable.
apemsel/attributed-string
551 Downloads
A collection of fast and easy to use attributed string classes for PHP. Attributed strings can have multiple attributes for each character of the string and are for example used for word processors and natural language processing.