Libraries tagged by tokenizer
hippophp/hippo
32 Downloads
Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.
devsrealm/tonics-template-system-extension-bundle
3724 Downloads
Custom Bundle Extension (Custom Tokenizers) for Tonics Template System
yuubit/tokenizer
412 Downloads
PHP Library to tokenize various inputs.
tanuel/tokenizer
15 Downloads
A tokenizer for PHP
robinthehood/tokenizer
975 Downloads
With Tokenizer you can easily convert a string to a stream or array of tokens.
ricwein/tokenizer
1634 Downloads
configurable string tokenizer
everstu/gpt3-tokenizer
345 Downloads
PHP package for Byte Pair Encoding (BPE) used by GPT-3.
sastrawi/tokenizer
947 Downloads
PHP library that allows you to tokenize Bahasa Indonesia.
tyhpdef/php-ext-tokenizer
6 Downloads
Tyhp type definitions for the PHP tokenizer extension (8.2+). Version-specific APIs are gated via declare(php=…) / #[\Tyhp\Php] (Story 20.5).
tsukasa/php-tokenizer
24171 Downloads
Polyfill for php extension Tokenizer
kevintweber/html-tokenizer
47765 Downloads
Will tokenize HTML.
yaroslavpopovic/laravel-text-chunker
92 Downloads
Flexible, strategy-based text chunking for Laravel — fork of droath/laravel-text-chunker with Laravel 13 support.
teepluss/wordbreaker
26 Downloads
Word breaker is Thai wording separator for using with fulltext software.
phpwatch/phptoken-polyfill
19 Downloads
A polyfill for PHP 8.0's PhpToken class
patchwork/php-parser
30 Downloads
Fast and extensible PHP code analysis and transformation