Libraries tagged by tokenzie

Simple string tokenizer for lexical scanning operations

Go to Download

Tokenizer for PHP code

Go to Download

tokenize sphinx configuration and that's all, folks.

Go to Download

Tokenizer for php

Go to Download

Will tokenize HTML.

Go to Download

Transform macro args to array.

Go to Download

Function parser for PHP functions, methods, and closures

Go to Download

A set of phplrt lexer abstractions

Go to Download

A program for performing lexical analysis, written in PHP

Go to Download

PHP 7.4 version of tiktoken

Go to Download

A PHP implementation of Complement Naive Bayes and SVM statistical classifiers, including a structure for building other classifier, multiple data sources and multiple caching backends

Go to Download

GPT-3-Encoder is a PHP implementation of OpenAI's original python encoder/decoder.

Go to Download

A tokenizer for Doctrine annotations.

Go to Download

Provides an infrastructure to tokenize content into nodes

Go to Download

Modernized, composer compatible version of Tim Whitlock's JTokenizer PHP Javascript parser and tokenizer library.

Go to Download