Libraries tagged by tokeniser
turanjanin/serbian-language-tools
2022 Downloads
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.
tacoberu/bnf
2687 Downloads
Framework for building BNF like parsers.
popsul/php-token-reflection
77369 Downloads
Library emulating the PHP internal reflection using just the tokenized source code.
hybridlogic/classifier
1445 Downloads
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.
didix16/php-grammar
7420 Downloads
A simple library for make Lexer and Parsers to build a language
camspiers/statistical-classifier
36988 Downloads
A PHP implementation of Complement Naive Bayes and SVM statistical classifiers, including a structure for building other classifier, multiple data sources and multiple caching backends
apicart/fql
10065 Downloads
Filter Query Language
hippophp/hippo
32 Downloads
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
devsrealm/tonics-template-system-extension-bundle
408 Downloads
Custom Bundle Extension (Custom Tokenizers) for Tonics Template System
ze/tokenizer-gpt3
1472 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
tanuel/tokenizer
14 Downloads
A tokenizer for PHP
ricwein/tokenizer
1031 Downloads
configurable string tokenizer
osirisgate/tokenizer
19 Downloads
Osirisgate token management.
gollumsf/url-tokenizer-bundle
1674 Downloads
Simple url tokenizer
funivan/php-tokenizer
38455 Downloads
Wrapper around token_get_all. Easy to extract and modify php code.