Libraries tagged by tokenizer
danny50610/bpe-tokeniser
198485 Downloads
PHP port for openai/tiktoken (most)
coderevolutionplugins/gpt-3-encoder-php
36428 Downloads
PHP BPE Text Encoder/Decoder for GPT-2 / GPT-3
udan11/sql-parser
1094 Downloads
A validating SQL lexer and parser with a focus on MySQL dialect.
strscan/strscan
125724 Downloads
Simple string tokenizer for lexical scanning operations
semji/gpt-3-tokenizer-php
68882 Downloads
PHP Text Tokenizer for GPT models
ltd-beget/dns-zone-tokenizer
61569 Downloads
tokenize dns zone files and that's all, folks.
remorhaz/php-unilex
123238 Downloads
Unilex: lexical analyzer generator with Unicode support written in PHP
phplrt/lexer
126811 Downloads
The phplrt lexer library that can be used for scanners implementation using PCRE
phootwork/tokenizer
207904 Downloads
Tokenizer for PHP code
artemeon/tokenizer
62821 Downloads
Tokenizer for php
tsukasa/php-tokenizer
22440 Downloads
Polyfill for php extension Tokenizer
ltd-beget/sphinx-configuration-tokenizer
19132 Downloads
tokenize sphinx configuration and that's all, folks.
infinityloop-dev/graphpinator-tokenizer
23148 Downloads
Tokenizer subproject for GraPHPinator: GraphQL server implementation for PHP.
kevintweber/html-tokenizer
47020 Downloads
Will tokenize HTML.
h4kuna/latte-php-tokenizer
15621 Downloads
Transform macro args to array.