Libraries tagged by tokeniser

danny50610/bpe-tokeniser

6 Favers
113862 Downloads

PHP port for openai/tiktoken (most)

Go to Download


mehrab-wj/tiktoken-php

17 Favers
16957 Downloads

a clone of python tiktoken but for PHP! fast BPE tokeniser for use with OpenAI's models.

Go to Download


hexydec/tokenise

3 Favers
6425 Downloads

A program for performing lexical analysis, written in PHP

Go to Download


theseer/tokenizer

5164 Favers
543147589 Downloads

A small library for converting tokenized PHP source code into XML and potentially other formats

Go to Download


phpunit/php-token-stream

6455 Favers
360997225 Downloads

Wrapper around PHP's tokenizer extension.

Go to Download


phpmyadmin/sql-parser

458 Favers
32250399 Downloads

A validating SQL lexer and parser with a focus on MySQL dialect.

Go to Download


jeremeamia/superclosure

1715 Favers
58043058 Downloads

Serialize Closure objects, including their context and binding

Go to Download


nette/tokenizer

141 Favers
3929937 Downloads

Nette Tokenizer

Go to Download


yethee/tiktoken

101 Favers
641432 Downloads

PHP version of tiktoken

Go to Download


netgen/query-translator

199 Favers
1412571 Downloads

Query Translator is a search query translator with AST representation

Go to Download


spiral/tokenizer

17 Favers
3620247 Downloads

Static Analysis: Class and Invocation locators

Go to Download


rajentrivedi/tokenizer-x

69 Favers
52307 Downloads

TokenizerX calculates required tokens for given prompt

Go to Download


gioni06/gpt3-tokenizer

85 Favers
273403 Downloads

PHP package for Byte Pair Encoding (BPE) used by GPT-3.

Go to Download


nilportugues/sql-query-formatter

38 Favers
755828 Downloads

A very lightweight PHP class that reformats unreadable and computer-generated SQL query statements to human-friendly, readable text.

Go to Download


andrewsville/php-token-reflection

190 Favers
1687552 Downloads

Library emulating the PHP internal reflection using just the tokenized source code.

Go to Download


Next >>