Libraries tagged by tokenizer

coderevolutionplugins/gpt-3-encoder-php

94 Favers
29536 Downloads

PHP BPE Text Encoder/Decoder for GPT-2 / GPT-3

strscan/strscan

12 Favers
98846 Downloads

Simple string tokenizer for lexical scanning operations

funivan/php-tokenizer

12 Favers
37311 Downloads

Wrapper around token_get_all. Easy to extract and modify php code.

semji/gpt-3-tokenizer-php

4 Favers
29011 Downloads

PHP Text Tokenizer for GPT models

ltd-beget/dns-zone-tokenizer

6 Favers
52836 Downloads

tokenize dns zone files and that's all, folks.

remorhaz/php-unilex

2 Favers
91601 Downloads

Unilex: lexical analyzer generator with Unicode support written in PHP

phplrt/lexer-contracts

1 Favers
88380 Downloads

A set of phplrt lexer abstractions

phplrt/lexer

3 Favers
77676 Downloads

The phplrt lexer library that can be used for scanners implementation using PCRE

guttedgarden/tiktoken

3 Favers
5522 Downloads

PHP 7.4 version of tiktoken

danny50610/bpe-tokeniser

4 Favers
20504 Downloads

PHP port for openai/tiktoken (most)

camspiers/statistical-classifier

177 Favers
36834 Downloads

A PHP implementation of Complement Naive Bayes and SVM statistical classifiers, including a structure for building other classifier, multiple data sources and multiple caching backends

phootwork/tokenizer

2 Favers
204601 Downloads

Tokenizer for PHP code

artemeon/tokenizer

0 Favers
42329 Downloads

Tokenizer for php

tsukasa/php-tokenizer

2 Favers
19979 Downloads

Polyfill for php extension Tokenizer

ltd-beget/sphinx-configuration-tokenizer

3 Favers
14731 Downloads

tokenize sphinx configuration and that's all, folks.

<< Previous Next >>