Download the PHP package guttedgarden/tiktoken without Composer
On this page you can find all versions of the php package guttedgarden/tiktoken. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Please rate this library. Is it a good library?
Informations about the package tiktoken
tiktoken-php
Yethee's tiktoken library port for PHP 7.4 without the Symfony package
Installation
Usage
Cache
The encoder uses an external vocabularies, so caching is used by default to avoid performance issues.
By default, the directory for temporary files is used.
You can override the directory for cache via environment variable TIKTOKEN_CACHE_DIR
or use EncoderProvider::setVocabCache()
:
Disable cache
You can disable the cache, if there are reasons for this, in one of the following ways:
- Set an empty string for the environment variable
TIKTOKEN_CACHE_DIR
. - Programmatically:
Limitations
- Encoding for GPT-2 is not supported.
- Special tokens (like
<|endofprompt|>
) are not supported.
License
MIT
All versions of tiktoken with dependencies
PHP Build Version
Package Version
Requires
php Version
>=7.4
The package guttedgarden/tiktoken contains the following files
Loading the files please wait ....