Libraries tagged by tokenizer
jeremeamia/functionparser
178368 Downloads
Function parser for PHP functions, methods, and closures
danny50610/bpe-tokeniser
495168 Downloads
PHP port for openai/tiktoken (most)
coderevolutionplugins/gpt-3-encoder-php
52208 Downloads
PHP BPE Text Encoder/Decoder for GPT-2 / GPT-3
udan11/sql-parser
7820 Downloads
A validating SQL lexer and parser with a focus on MySQL dialect.
strscan/strscan
177950 Downloads
Simple string tokenizer for lexical scanning operations
ze/tokenizer-gpt3
39127 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
nette/tokenizer
Downloads
codewithkyrian/tokenizers
710 Downloads
Fast, pure-PHP tokenizer library compatible with Hugging Face tokenizers for encoding and decoding text
artemeon/tokenizer
93727 Downloads
Tokenizer for php
ltd-beget/dns-zone-tokenizer
262328 Downloads
tokenize dns zone files and that's all, folks.
simpletools/functionparser
58501 Downloads
Function parser for PHP functions, methods, and closures
remorhaz/php-unilex
176266 Downloads
Unilex: lexical analyzer generator with Unicode support written in PHP
phplrt/lexer-contracts
192045 Downloads
A set of phplrt lexer abstractions
phplrt/lexer-builder
5117 Downloads
The phplrt component for assembling a lexer
phplrt/lexer
178399 Downloads
The phplrt lexer library that can be used for scanners implementation using PCRE