Download the PHP package textualization/ropherta-tokenizer without Composer
On this page you can find all versions of the php package textualization/ropherta-tokenizer. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Table of contents
Download textualization/ropherta-tokenizer
More information about textualization/ropherta-tokenizer
Files in textualization/ropherta-tokenizer
Download textualization/ropherta-tokenizer
More information about textualization/ropherta-tokenizer
Files in textualization/ropherta-tokenizer
Vendor textualization
Package ropherta-tokenizer
Short Description GPT3Tokenizer (BPE) with Roberta-base vocabulary.
License Apache-2.0
Package ropherta-tokenizer
Short Description GPT3Tokenizer (BPE) with Roberta-base vocabulary.
License Apache-2.0
Please rate this library. Is it a good library?
Informations about the package ropherta-tokenizer
BPE Tokenizer for Ropherta (subclass of GPT3Tokenizer)
This is just a wrapper around GPT3Tokenizer using the HuggingFace RoBERTa vocab and merge files.
See GPT3 documentation for example use (or the generated test case under tests/
).
XLM Tokenizer
To use the multilingual version, the SentencePiece dependency needs to be initialized and an aditional model file needs to be downloaded:
Sponsors
We thank our sponsor:
All versions of ropherta-tokenizer with dependencies
PHP Build Version
Package Version
The package textualization/ropherta-tokenizer contains the following files
Loading the files please wait ....