PHP code example of rajentrivedi / tokenizer-x

1. Go to this page and download the library: Download rajentrivedi/tokenizer-x library. Choose the download type require.

2. Extract the ZIP file and open the index.php.

3. Add this code to the index.php.

    
        
<?php
require_once('vendor/autoload.php');

/* Start to develop here. Best regards https://php-download.com/ */

rajentrivedi / tokenizer-x example snippets


use Rajentrivedi\TokenizerX\TokenizerX;
TokenizerX::count("how are you?");


use Rajentrivedi\TokenizerX\TokenizerX;
TokenizerX::count("how are you?", "gpt-4");


TokenizerX::count(file_get_contents('path_to_file'));


TokenizerX::tokens(file_get_contents('path_to_file'));