Libraries tagged by tokeniser
kerasai/behat-tokenizer
20413 Downloads
Token replacement for Behat test suite.
aisk/mistral-tokenizer
119 Downloads
PHP implementation of Mistral's tokenizers
turanjanin/serbian-language-tools
2053 Downloads
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.
tacoberu/bnf
2756 Downloads
Framework for building BNF like parsers.
popsul/php-token-reflection
77501 Downloads
Library emulating the PHP internal reflection using just the tokenized source code.
hybridlogic/classifier
1445 Downloads
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.
camspiers/statistical-classifier
36991 Downloads
A PHP implementation of Complement Naive Bayes and SVM statistical classifiers, including a structure for building other classifier, multiple data sources and multiple caching backends
apicart/fql
10124 Downloads
Filter Query Language
hippophp/hippo
32 Downloads
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
devsrealm/tonics-template-system-extension-bundle
636 Downloads
Custom Bundle Extension (Custom Tokenizers) for Tonics Template System
textualization/ropherta-tokenizer
91 Downloads
GPT3Tokenizer (BPE) with Roberta-base vocabulary.
tanuel/tokenizer
15 Downloads
A tokenizer for PHP
mathematicator-core/tokenizer
4665 Downloads
Math Tokenizer
interitty/tokenizer
52 Downloads
Use regular expressions to split a given string into tokens.
gollumsf/url-tokenizer-bundle
1701 Downloads
Simple url tokenizer