Libraries tagged by tokenizer
danny50610/bpe-tokeniser
122132 Downloads
PHP port for openai/tiktoken (most)
coderevolutionplugins/gpt-3-encoder-php
34288 Downloads
PHP BPE Text Encoder/Decoder for GPT-2 / GPT-3
strscan/strscan
116759 Downloads
Simple string tokenizer for lexical scanning operations
semji/gpt-3-tokenizer-php
49495 Downloads
PHP Text Tokenizer for GPT models
ltd-beget/dns-zone-tokenizer
58227 Downloads
tokenize dns zone files and that's all, folks.
remorhaz/php-unilex
110806 Downloads
Unilex: lexical analyzer generator with Unicode support written in PHP
phplrt/lexer-contracts
121227 Downloads
A set of phplrt lexer abstractions
phplrt/lexer
109966 Downloads
The phplrt lexer library that can be used for scanners implementation using PCRE
phootwork/tokenizer
206840 Downloads
Tokenizer for PHP code
artemeon/tokenizer
55284 Downloads
Tokenizer for php
tsukasa/php-tokenizer
21490 Downloads
Polyfill for php extension Tokenizer
ltd-beget/sphinx-configuration-tokenizer
17893 Downloads
tokenize sphinx configuration and that's all, folks.
infinityloop-dev/graphpinator-tokenizer
20622 Downloads
Tokenizer subproject for GraPHPinator: GraphQL server implementation for PHP.
kevintweber/html-tokenizer
46326 Downloads
Will tokenize HTML.
h4kuna/latte-php-tokenizer
12992 Downloads
Transform macro args to array.