Libraries tagged by tokenize
ze/tokenizer-gpt3
1456 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
tanuel/tokenizer
14 Downloads
A tokenizer for PHP
ricwein/tokenizer
1014 Downloads
configurable string tokenizer
irfan-chowdhury/bkash-tokenized-checkout
25 Downloads
This is a bkash tokenized payment gateway solution with laravel. Integrate with the payment solution that fits all business needs and Enrich the customers' online payment experience.
hippophp/tokenizer
1105 Downloads
Wrapper around token_get_all
funivan/php-tokenizer
38446 Downloads
Wrapper around token_get_all. Easy to extract and modify php code.
everstu/gpt3-tokenizer
121 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
yannickl88/css-tokenizer
30833 Downloads
CSS tokenizer
sastrawi/tokenizer
946 Downloads
PHP library that allows you to tokenize Bahasa Indonesia.
vanderlee/comprehend
16 Downloads
Framework for building BNF LR(1) parsers
teepluss/wordbreaker
25 Downloads
Word breaker is Thai wording separator for using with fulltext software.
phpwatch/phptoken-polyfill
17 Downloads
A polyfill for PHP 8.0's PhpToken class
patchwork/php-parser
26 Downloads
Fast and extensible PHP code analysis and transformation
dbeurive/lexer
2343 Downloads
This project implements a simple lexer.
crazzy501/gpt-3-encoder-php
320 Downloads
PHP BPE Text Encoder/Decoder for GPT-2 / GPT-3. Based on coderevolutionplugins/gpt-3-encoder-php