Libraries tagged by tokenized
h4kuna/latte-php-tokenizer
11945 Downloads
Transform macro args to array.
tacoberu/bnf
2253 Downloads
Framework for building BNF like parsers.
hybridlogic/classifier
1431 Downloads
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.
hippophp/hippo
32 Downloads
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
didix16/php-grammar
6076 Downloads
A simple library for make Lexer and Parsers to build a language
tanuel/tokenizer
14 Downloads
A tokenizer for PHP
mathematicator-core/tokenizer
4412 Downloads
Math Tokenizer
interitty/tokenizer
38 Downloads
Use regular expressions to split a given string into tokens.
hippophp/tokenizer
1101 Downloads
Wrapper around token_get_all
everstu/gpt3-tokenizer
59 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
sastrawi/tokenizer
946 Downloads
PHP library that allows you to tokenize Bahasa Indonesia.
xtreed/php-token-stream
3520 Downloads
Wrapper around PHP's tokenizer extension.
vanderlee/comprehend
14 Downloads
Framework for building BNF LR(1) parsers
turanjanin/serbian-language-tools
1918 Downloads
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.
teepluss/wordbreaker
23 Downloads
Word breaker is Thai wording separator for using with fulltext software.