Libraries tagged by tokeniser

danny50610/bpe-tokeniser

6 Favers
122911 Downloads

PHP port for openai/tiktoken (most)

Go to Download


mehrab-wj/tiktoken-php

17 Favers
18253 Downloads

a clone of python tiktoken but for PHP! fast BPE tokeniser for use with OpenAI's models.

Go to Download


hexydec/tokenise

3 Favers
6589 Downloads

A program for performing lexical analysis, written in PHP

Go to Download


theseer/tokenizer

5166 Favers
550434834 Downloads

A small library for converting tokenized PHP source code into XML and potentially other formats

Go to Download


phpunit/php-token-stream

6455 Favers
362113919 Downloads

Wrapper around PHP's tokenizer extension.

Go to Download


phpmyadmin/sql-parser

457 Favers
33392336 Downloads

A validating SQL lexer and parser with a focus on MySQL dialect.

Go to Download


jeremeamia/superclosure

1715 Favers
58141404 Downloads

Serialize Closure objects, including their context and binding

Go to Download


nette/tokenizer

141 Favers
3986448 Downloads

Nette Tokenizer

Go to Download


yethee/tiktoken

103 Favers
688106 Downloads

PHP version of tiktoken

Go to Download


netgen/query-translator

200 Favers
1427996 Downloads

Query Translator is a search query translator with AST representation

Go to Download


spiral/tokenizer

17 Favers
3728285 Downloads

Static Analysis: Class and Invocation locators

Go to Download


rajentrivedi/tokenizer-x

71 Favers
56051 Downloads

TokenizerX calculates required tokens for given prompt

Go to Download


gioni06/gpt3-tokenizer

85 Favers
283644 Downloads

PHP package for Byte Pair Encoding (BPE) used by GPT-3.

Go to Download


nilportugues/sql-query-formatter

38 Favers
769091 Downloads

A very lightweight PHP class that reformats unreadable and computer-generated SQL query statements to human-friendly, readable text.

Go to Download


andrewsville/php-token-reflection

190 Favers
1691277 Downloads

Library emulating the PHP internal reflection using just the tokenized source code.

Go to Download


Next >>