Libraries tagged by tokenized
ecomdev/magento-translate-tool
6 Downloads
Magento translation tool that utilizes PHP Tokenizer in order to generate csv files from PHP, XML, PHTML code
hipay/hipay-enterprise-sdk-js
1932 Downloads
The HiPay Fullservice SDK for JavaScript allows you to tokenize credit or debit cards against the HiPay Fullservice payment platform.
yuubit/tokenizer
403 Downloads
PHP Library to tokenize various inputs.
your1/tokenizer-x
273 Downloads
TokenizerX calculates required tokens for given prompt
yep/tokenizer
8 Downloads
Tokenizer
wpscholar/php-tokenizer
3 Downloads
An experiment with using the PHP tokenizer functions.
vanry/tokenizer
2 Downloads
支持 scws, phpanalysis 和 jieba 的中文分词器
ueef/tokenizer
84 Downloads
ueef/auth-tokenizer
901 Downloads
thruster/tokenizer
6 Downloads
Thruster Tokenizer Component
textualization/ropherta-tokenizer
91 Downloads
GPT3Tokenizer (BPE) with Roberta-base vocabulary.
stella-maris/tokenizer
0 Downloads
A generic Tokenizer
soloproyectos-php/text-tokenizer
891 Downloads
A library to split a string into smaller tokens. This library is part of the SoloProyectos PHP API.
sincco/tokenizer
180 Downloads
Create and validate token for user data
scripturadesign/tokenizer
7 Downloads
Tokenizer