Download the PHP package rajentrivedi/tokenizer-x without Composer

On this page you can find all versions of the php package rajentrivedi/tokenizer-x. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.

FAQ

After the download, you have to make one include require_once('vendor/autoload.php');. After that you have to import the classes with use statements.

Example:
If you use only one package a project is not needed. But if you use more then one package, without a project it is not possible to import the classes with use statements.

In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
Some PHP packages are not free to download and because of that hosted in private repositories. In this case some credentials are needed to access such packages. Please use the auth.json textarea to insert credentials, if a package is coming from a private repository. You can look here for more information.

  • Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
  • To use Composer is sometimes complicated. Especially for beginners.
  • Composer needs much resources. Sometimes they are not available on a simple webspace.
  • If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
  • Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Please rate this library. Is it a good library?

Example code of rajentrivedi/tokenizer-x

Informations about the package tokenizer-x

Calculates the OpenAI tokens for the given prompt

Latest Version on Packagist

GitHub Code Style Action Status Total Downloads

TokenizerX supports Laravel 11 and Laravel 10.

Installation

TokenizerX

TokenizerX is a Laravel package designed to streamline tokenization processes in your applications. With the latest update, TokenizerX now supports cutting-edge GPT-4 models, providing advanced natural language processing capabilities.

It calculates the tokens required for a given prompt before requesting the OpenAI REST API. This package helps to ensure that the user does not exceed the OpenAI API token limit and can generate accurate responses.

To access the OpenAI Rest API, you may consider the beautiful Laravel Package OpenAI PHP.

Supported OpenAI Models

Supported Encoding

Installation

You can install the package via composer:

Usage

By default package will consider GPT-3 model

If you want token counts for specific OpenAI model, you can pass model as a second argument from above given supported model list.

You can also read the text from file

Please make sure that text of the file don't change while reading the file programmatically, this may happen due to encoding. You can check the generated token IDs by using following

This will return an array of tokens generated & compare those token IDs with OpenAI Tokenizer

You can also use the OpenAI Tokenizer to double-check package generated token counts.

Support

If you find TokenizerX helpful and would like to support its ongoing development, you can contribute by buying me a coffee! Your support helps in maintaining and improving the package for the Laravel community.

ko-fi

Testing

Changelog

Please see CHANGELOG for more information on what has changed recently.

Contributing

Please see CONTRIBUTING for details.

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities.

⭐ Star the Repository ⭐

If you find this project useful or interesting, I kindly request you to give it a ⭐ star on GitHub. Your support will encourage and motivate me to continue improving and maintaining this project.

By starring the repository, you can show appreciation for the work put into developing this open-source project. It also helps to increase its visibility, making it more accessible to other developers and potentially attracting contributors.

To give a ⭐ star, simply click on the Star button at the top-right corner of the repository page.

Credits

License

TokenizerX is developed using

The MIT License (MIT). Please see License File for more information.

All versions of tokenizer-x with dependencies

PHP Build Version
Package Version
Requires php Version ^8.1
illuminate/console Version ^9.52.16 || ^10.28.0 || ^11.0
illuminate/contracts Version ^10.0 || ^11.0
illuminate/support Version ^9.52.16 || ^10.28.0 || ^11.0
spatie/laravel-package-tools Version ^1.14.0
yethee/tiktoken Version ^0.5.1
Composer command for our command line client (download client) This client runs in each environment. You don't need a specific PHP version etc. The first 20 API calls are free. Standard composer command

The package rajentrivedi/tokenizer-x contains the following files

Loading the files please wait ....