Libraries tagged by tokenized
hexydec/tokenise
12195 Downloads
A program for performing lexical analysis, written in PHP
guttedgarden/tiktoken
22930 Downloads
PHP 7.4 version of tiktoken
globus-studio/php-obfuscator
91 Downloads
A token-based PHP source code obfuscator with full support for PHP 8.3 through 8.5.
frolaxhq/laravel-llm-tokenkit
806 Downloads
Token estimation, cost calculation & context-window utilities for LLM-powered Laravel apps
rahul900day/gpt-3-encoder
32454 Downloads
GPT-3-Encoder is a PHP implementation of OpenAI's original python encoder/decoder.
mwstake/mediawiki-lib-nodes
31041 Downloads
Provides an infrastructure to tokenize content into nodes
textualization/ropherta-tokenizer
633 Downloads
GPT3Tokenizer (BPE) with Roberta-base vocabulary.
ride/lib-tokenizer
8076 Downloads
Tokenizer library of the Ride framework.
ricwein/tokenizer
1553 Downloads
configurable string tokenizer
ltd-beget/sphinx-configuration-tokenizer
23972 Downloads
tokenize sphinx configuration and that's all, folks.
kerasai/behat-tokenizer
21138 Downloads
Token replacement for Behat test suite.
codewithkyrian/tokenizers
35 Downloads
Fast, pure-PHP tokenizer library compatible with Hugging Face tokenizers for encoding and decoding text
yannickl88/css-tokenizer
30945 Downloads
CSS tokenizer
hippophp/tokenizer
1311 Downloads
Wrapper around token_get_all
turanjanin/serbian-language-tools
2358 Downloads
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.