Libraries tagged by tokeniser
funivan/php-tokenizer
38455 Downloads
Wrapper around token_get_all. Easy to extract and modify php code.
everstu/gpt3-tokenizer
169 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
yannickl88/css-tokenizer
30839 Downloads
CSS tokenizer
sastrawi/tokenizer
947 Downloads
PHP library that allows you to tokenize Bahasa Indonesia.
hippophp/tokenizer
1111 Downloads
Wrapper around token_get_all
vanderlee/comprehend
17 Downloads
Framework for building BNF LR(1) parsers
teepluss/wordbreaker
26 Downloads
Word breaker is Thai wording separator for using with fulltext software.
taufik-nurrohman/minify
6 Downloads
Reduce CSS, HTML, JavaScript, JSON, PHP, and XML code size on the server side.
rudl/logger
2726 Downloads
Rudl UDP cluster logging library (PSR4 compliant)
phpwatch/phptoken-polyfill
18 Downloads
A polyfill for PHP 8.0's PhpToken class
patchwork/php-parser
26 Downloads
Fast and extensible PHP code analysis and transformation
didix16/php-grammar
7665 Downloads
A simple library for make Lexer and Parsers to build a language
dbeurive/lexer
2413 Downloads
This project implements a simple lexer.
chrismcmacken/phptools
3558 Downloads
Small utilities that assist in developing applications
masnathan/sql-query-formatter
1792 Downloads
A very lightweight PHP class that reformats unreadable and computer-generated SQL query statements to human-friendly, readable text.