Download the PHP package webrek/tokenizers without Composer
On this page you can find all versions of the php package webrek/tokenizers. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Table of contents
Download webrek/tokenizers
More information about webrek/tokenizers
Files in webrek/tokenizers
Download webrek/tokenizers
More information about webrek/tokenizers
Files in webrek/tokenizers
Vendor webrek
Package tokenizers
Short Description Native byte-level BPE (tiktoken-compatible: cl100k_base/o200k_base) plus WordPiece and Unigram tokenizers for PHP 8.3+, with a process-shared vocab cache. Loads HuggingFace tokenizer.json models.
License Apache-2.0
Package tokenizers
Short Description Native byte-level BPE (tiktoken-compatible: cl100k_base/o200k_base) plus WordPiece and Unigram tokenizers for PHP 8.3+, with a process-shared vocab cache. Loads HuggingFace tokenizer.json models.
License Apache-2.0
Please rate this library. Is it a good library?
Informations about the package tokenizers
All versions of tokenizers with dependencies
PHP Build Version
Package Version
Requires
php Version
>=8.3
The package webrek/tokenizers contains the following files
Loading the files please wait ...