Libraries tagged by tokenize
guttedgarden/tiktoken
22897 Downloads
PHP 7.4 version of tiktoken
globus-studio/php-obfuscator
82 Downloads
A token-based PHP source code obfuscator with full support for PHP 8.3 through 8.5.
frolaxhq/laravel-llm-tokenkit
748 Downloads
Token estimation, cost calculation & context-window utilities for LLM-powered Laravel apps
rahul900day/gpt-3-encoder
32416 Downloads
GPT-3-Encoder is a PHP implementation of OpenAI's original python encoder/decoder.
mwstake/mediawiki-lib-nodes
30950 Downloads
Provides an infrastructure to tokenize content into nodes
ltd-beget/sphinx-configuration-tokenizer
23971 Downloads
tokenize sphinx configuration and that's all, folks.
textualization/ropherta-tokenizer
627 Downloads
GPT3Tokenizer (BPE) with Roberta-base vocabulary.
ride/lib-tokenizer
8074 Downloads
Tokenizer library of the Ride framework.
ricwein/tokenizer
1553 Downloads
configurable string tokenizer
kerasai/behat-tokenizer
21138 Downloads
Token replacement for Behat test suite.
codewithkyrian/tokenizers
35 Downloads
Fast, pure-PHP tokenizer library compatible with Hugging Face tokenizers for encoding and decoding text
yannickl88/css-tokenizer
30939 Downloads
CSS tokenizer
hippophp/tokenizer
1311 Downloads
Wrapper around token_get_all
turanjanin/serbian-language-tools
2358 Downloads
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.
tacoberu/bnf
3257 Downloads
Framework for building BNF like parsers.