Libraries tagged by tokeniser
everstu/gpt3-tokenizer
132 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
averay/tokenised-strings
248 Downloads
A lightweight system for rendering tokens in templated strings.
yannickl88/css-tokenizer
30838 Downloads
CSS tokenizer
sastrawi/tokenizer
946 Downloads
PHP library that allows you to tokenize Bahasa Indonesia.
hippophp/tokenizer
1106 Downloads
Wrapper around token_get_all
vanderlee/comprehend
16 Downloads
Framework for building BNF LR(1) parsers
teepluss/wordbreaker
25 Downloads
Word breaker is Thai wording separator for using with fulltext software.
rudl/logger
2666 Downloads
Rudl UDP cluster logging library (PSR4 compliant)
phpwatch/phptoken-polyfill
17 Downloads
A polyfill for PHP 8.0's PhpToken class
patchwork/php-parser
26 Downloads
Fast and extensible PHP code analysis and transformation
dbeurive/lexer
2393 Downloads
This project implements a simple lexer.
cxrxexo/tiktoken
2092 Downloads
PHP version of tiktoken. PHP 7.4+ required.
crazzy501/gpt-3-encoder-php
379 Downloads
PHP BPE Text Encoder/Decoder for GPT-2 / GPT-3. Based on coderevolutionplugins/gpt-3-encoder-php
chrismcmacken/phptools
3558 Downloads
Small utilities that assist in developing applications
masnathan/sql-query-formatter
1765 Downloads
A very lightweight PHP class that reformats unreadable and computer-generated SQL query statements to human-friendly, readable text.