Libraries tagged by tokenzie
textualization/ropherta-tokenizer
85 Downloads
GPT3Tokenizer (BPE) with Roberta-base vocabulary.
tanuel/tokenizer
15 Downloads
A tokenizer for PHP
ricwein/tokenizer
1055 Downloads
configurable string tokenizer
irfan-chowdhury/bkash-tokenized-checkout
28 Downloads
This is a bkash tokenized payment gateway solution with laravel. Integrate with the payment solution that fits all business needs and Enrich the customers' online payment experience.
funivan/php-tokenizer
38455 Downloads
Wrapper around token_get_all. Easy to extract and modify php code.
everstu/gpt3-tokenizer
138 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
aisk/mistral-tokenizer
55 Downloads
PHP implementation of Mistral's tokenizers
yannickl88/css-tokenizer
30839 Downloads
CSS tokenizer
sastrawi/tokenizer
947 Downloads
PHP library that allows you to tokenize Bahasa Indonesia.
vanderlee/comprehend
17 Downloads
Framework for building BNF LR(1) parsers
teepluss/wordbreaker
26 Downloads
Word breaker is Thai wording separator for using with fulltext software.
rudl/logger
2694 Downloads
Rudl UDP cluster logging library (PSR4 compliant)
phpwatch/phptoken-polyfill
18 Downloads
A polyfill for PHP 8.0's PhpToken class
patchwork/php-parser
26 Downloads
Fast and extensible PHP code analysis and transformation
dbeurive/lexer
2398 Downloads
This project implements a simple lexer.