Libraries tagged by tokeniser
dbeurive/lexer
2753 Downloads
This project implements a simple lexer.
crazzy501/gpt-3-encoder-php
421 Downloads
PHP BPE Text Encoder/Decoder for GPT-2 / GPT-3. Based on coderevolutionplugins/gpt-3-encoder-php
camspiers/statistical-classifier
37020 Downloads
A PHP implementation of Complement Naive Bayes and SVM statistical classifiers, including a structure for building other classifier, multiple data sources and multiple caching backends
aeliot/yaml-token
154 Downloads
Tokenizer of YAML
hippophp/hippo
32 Downloads
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
devsrealm/tonics-template-system-extension-bundle
3274 Downloads
Custom Bundle Extension (Custom Tokenizers) for Tonics Template System
mistralys/jtokenizer
7932 Downloads
Modernized, composer compatible version of Tim Whitlock's JTokenizer PHP Javascript parser and tokenizer library.
tsukasa/php-tokenizer
24138 Downloads
Polyfill for php extension Tokenizer
tanuel/tokenizer
15 Downloads
A tokenizer for PHP
robinthehood/tokenizer
949 Downloads
With Tokenizer you can easily convert a string to a stream or array of tokens.
ricwein/tokenizer
1553 Downloads
configurable string tokenizer
funivan/php-tokenizer
38456 Downloads
Wrapper around token_get_all. Easy to extract and modify php code.
componenta/tokenizer
9 Downloads
Token-based PHP source inspection helpers for Componenta
sastrawi/tokenizer
947 Downloads
PHP library that allows you to tokenize Bahasa Indonesia.
kevintweber/html-tokenizer
47765 Downloads
Will tokenize HTML.