PHP code example of mathsgod / token-text-splitter

1. Go to this page and download the library: Download mathsgod/token-text-splitter library. Choose the download type require.

2. Extract the ZIP file and open the index.php.

3. Add this code to the index.php.
    
        
<?php
require_once('vendor/autoload.php');

/* Start to develop here. Best regards https://php-download.com/ */

    

mathsgod / token-text-splitter example snippets



use TextSplitter\TokenTextSplitter;

$text = "蘋果公司(Apple Inc.)是美國的一家跨國科技公司,總部位於加利福尼亞州的庫比蒂諾。蘋果公司的硬體產品包括iPhone智慧型手機、iPad平板電腦、Mac個人電腦、iPod多媒體播放器、Apple Watch智慧手錶和Apple TV數位媒體機。蘋果公司的軟體產品包括iOS、iPadOS、macOS、watchOS和tvOS作業系統,iTunes多媒體播放軟體,Safari網頁瀏覽器,iLife和iWork生產力套件,Final Cut Pro X和Logic Pro X專業影音剪輯軟體。蘋果公司的線上服務包括App Store、Apple Music、iCloud、iTunes Store和Apple TV+。蘋果公司的零售店面遍佈全球,是全球最大的科技公司之一。";

// token size is 10
// overlap is 5
$splitter = new TextSplitter\TokenTextSplitter("gpt-4o", 10, 5);

$chunks = $splitter->splitText($text);

print_r($chunks);