Libraries tagged by tokenize
ricwein/tokenizer
1153 Downloads
configurable string tokenizer
kerasai/behat-tokenizer
20331 Downloads
Token replacement for Behat test suite.
aisk/mistral-tokenizer
97 Downloads
PHP implementation of Mistral's tokenizers
turanjanin/serbian-language-tools
2050 Downloads
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.
tacoberu/bnf
2755 Downloads
Framework for building BNF like parsers.
popsul/php-token-reflection
77499 Downloads
Library emulating the PHP internal reflection using just the tokenized source code.
hybridlogic/classifier
1445 Downloads
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.
camspiers/statistical-classifier
36991 Downloads
A PHP implementation of Complement Naive Bayes and SVM statistical classifiers, including a structure for building other classifier, multiple data sources and multiple caching backends
apicart/fql
10124 Downloads
Filter Query Language
hippophp/hippo
32 Downloads
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
herrera-io/annotations
127559 Downloads
A tokenizer for Doctrine annotations.
laurynasgadl/travers
458015 Downloads
Travers - PHP array traversing via tokenized paths
devsrealm/tonics-template-system-extension-bundle
620 Downloads
Custom Bundle Extension (Custom Tokenizers) for Tonics Template System
ze/tokenizer-gpt3
1490 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
textualization/ropherta-tokenizer
91 Downloads
GPT3Tokenizer (BPE) with Roberta-base vocabulary.