Libraries tagged by tokenizer
fi1a/tokenizer
16628 Downloads
Лексический анализатор строк (tokenizer). Разбор входной последовательности символов на распознанные группы (лексемы).
artemeon/tokenizer
87501 Downloads
Tokenizer for php
infinityloop-dev/graphpinator-tokenizer
42815 Downloads
Tokenizer subproject for GraPHPinator: GraphQL server implementation for PHP.
aisk/mistral-tokenizer
18268 Downloads
PHP implementation of Mistral's tokenizers
h4kuna/latte-php-tokenizer
25542 Downloads
Transform macro args to array.
simpletools/functionparser
48610 Downloads
Function parser for PHP functions, methods, and closures
phplrt/lexer-contracts
177586 Downloads
A set of phplrt lexer abstractions
hexydec/tokenise
12216 Downloads
A program for performing lexical analysis, written in PHP
guttedgarden/tiktoken
23057 Downloads
PHP 7.4 version of tiktoken
globus-studio/php-obfuscator
101 Downloads
A token-based PHP source code obfuscator with full support for PHP 8.3 through 8.5.
frolaxhq/laravel-llm-tokenkit
887 Downloads
Token estimation, cost calculation & context-window utilities for LLM-powered Laravel apps
rahul900day/gpt-3-encoder
32483 Downloads
GPT-3-Encoder is a PHP implementation of OpenAI's original python encoder/decoder.
codewithkyrian/tokenizers
37 Downloads
Fast, pure-PHP tokenizer library compatible with Hugging Face tokenizers for encoding and decoding text
hippophp/tokenizer
1311 Downloads
Wrapper around token_get_all
textualization/ropherta-tokenizer
645 Downloads
GPT3Tokenizer (BPE) with Roberta-base vocabulary.