Libraries tagged by tokenized
hippophp/hippo
32 Downloads
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
didix16/php-grammar
6544 Downloads
A simple library for make Lexer and Parsers to build a language
cxrxexo/tiktoken
2049 Downloads
PHP version of tiktoken. PHP 7.4+ required.
camspiers/statistical-classifier
36966 Downloads
A PHP implementation of Complement Naive Bayes and SVM statistical classifiers, including a structure for building other classifier, multiple data sources and multiple caching backends
ze/tokenizer-gpt3
1418 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
tanuel/tokenizer
14 Downloads
A tokenizer for PHP
hippophp/tokenizer
1101 Downloads
Wrapper around token_get_all
gollumsf/url-tokenizer-bundle
1496 Downloads
Simple url tokenizer
funivan/php-tokenizer
38440 Downloads
Wrapper around token_get_all. Easy to extract and modify php code.
everstu/gpt3-tokenizer
98 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
averay/tokenised-strings
219 Downloads
A lightweight system for rendering tokens in templated strings.
sastrawi/tokenizer
946 Downloads
PHP library that allows you to tokenize Bahasa Indonesia.
yurunsoft/tiktoken
212 Downloads
PHP version of tiktoken
vanderlee/comprehend
16 Downloads
Framework for building BNF LR(1) parsers
turanjanin/serbian-language-tools
1933 Downloads
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.