Libraries tagged by tokenized
mathematicator-core/tokenizer
4675 Downloads
Math Tokenizer
kerasai/behat-tokenizer
20485 Downloads
Token replacement for Behat test suite.
tacoberu/bnf
2780 Downloads
Framework for building BNF like parsers.
hybridlogic/classifier
1445 Downloads
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.
hosmelq/search-syntax-parser
364 Downloads
Parse search queries into structured data with field searches, boolean logic, ranges, lists, and adapters.
didix16/php-grammar
7874 Downloads
A simple library for make Lexer and Parsers to build a language
dbeurive/lexer
2430 Downloads
This project implements a simple lexer.
hippophp/hippo
32 Downloads
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
devsrealm/tonics-template-system-extension-bundle
928 Downloads
Custom Bundle Extension (Custom Tokenizers) for Tonics Template System
tanuel/tokenizer
15 Downloads
A tokenizer for PHP
ricwein/tokenizer
1180 Downloads
configurable string tokenizer
gollumsf/url-tokenizer-bundle
1720 Downloads
Simple url tokenizer
funivan/php-tokenizer
38456 Downloads
Wrapper around token_get_all. Easy to extract and modify php code.
everstu/gpt3-tokenizer
178 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
averay/tokenised-strings
256 Downloads
A lightweight system for rendering tokens in templated strings.