Libraries tagged by tokenizer
textualization/ropherta-tokenizer
62 Downloads
GPT3Tokenizer (BPE) with Roberta-base vocabulary.
phpfacile/string-tokenizer
39 Downloads
Library to parse a string (especially to extract text and keywords identified by a begin and end delimiter and to substitute keywords by values)
gollumsf/url-tokenizer-bundle
1469 Downloads
Simple url tokenizer
vanderlee/comprehend
16 Downloads
Framework for building BNF LR(1) parsers
teepluss/wordbreaker
25 Downloads
Word breaker is Thai wording separator for using with fulltext software.
rudl/logger
2382 Downloads
Rudl UDP cluster logging library (PSR4 compliant)
phpwatch/phptoken-polyfill
17 Downloads
A polyfill for PHP 8.0's PhpToken class
paulorlima9/php-token-stream
11 Downloads
Wrapper around PHP's tokenizer extension.
patchwork/php-parser
26 Downloads
Fast and extensible PHP code analysis and transformation
html5/htmlreader
3252 Downloads
Html5 stream tokenizer/reader (not using libxml)
dbeurive/lexer
2287 Downloads
This project implements a simple lexer.
chrismcmacken/phptools
3558 Downloads
Small utilities that assist in developing applications
antto1/superclosure
152 Downloads
Serialize Closure objects, including their context and binding
masnathan/sql-query-formatter
1700 Downloads
A very lightweight PHP class that reformats unreadable and computer-generated SQL query statements to human-friendly, readable text.
crazzy501/gpt-3-encoder-php
297 Downloads
PHP BPE Text Encoder/Decoder for GPT-2 / GPT-3. Based on coderevolutionplugins/gpt-3-encoder-php