Libraries tagged by tokenize
codechap/yii3-context-trimmer
20 Downloads
Tokenizer-agnostic text preprocessor for trimming and optimising content for LLM context windows.
chrismcmacken/phptools
3558 Downloads
Small utilities that assist in developing applications
robo47/php-manipulator
8 Downloads
Manipulation of php code, automated refactoring based on phps tokenizer
nvl/mail-notifications
14 Downloads
Provider-neutral Laravel mail tracking, scheduling, MailerSend integration, and tokenized presentation
ecomdev/magento-translate-tool
6 Downloads
Magento translation tool that utilizes PHP Tokenizer in order to generate csv files from PHP, XML, PHTML code
hipay/hipay-enterprise-sdk-js
1932 Downloads
The HiPay Fullservice SDK for JavaScript allows you to tokenize credit or debit cards against the HiPay Fullservice payment platform.
philwaters/tokenizer
12 Downloads
Simple wrapper class around PHP's built in tokenize functions'
org_heigl/tokenizer
2266 Downloads
Provides a way of tokenizing strings
dennislindsey/tokenize
24 Downloads
Tokenization Interface
zen0x7/tokenizer
22 Downloads
PHP SDK for Service Framework authentication and token generation.
yuubit/tokenizer
403 Downloads
PHP Library to tokenize various inputs.
your1/tokenizer-x
273 Downloads
TokenizerX calculates required tokens for given prompt
yep/tokenizer
8 Downloads
Tokenizer
wpscholar/php-tokenizer
3 Downloads
An experiment with using the PHP tokenizer functions.
webrek/tokenizers
0 Downloads
Native byte-level BPE (tiktoken-compatible: cl100k_base/o200k_base) plus WordPiece and Unigram tokenizers for PHP 8.3+, with a process-shared vocab cache. Loads HuggingFace tokenizer.json models and counts Claude/Gemini tokens via their APIs.