Libraries tagged by tokenizer

Wrapper around token_get_all

Go to Download

GPT3Tokenizer (BPE) with Roberta-base vocabulary.

Go to Download

Tokenizer library of the Ride framework.

Go to Download

tokenize sphinx configuration and that's all, folks.

Go to Download

Token replacement for Behat test suite.

Go to Download

CSS tokenizer

Go to Download

PHP version of tiktoken

Go to Download

Framework for building BNF like parsers.

Go to Download

Pure PHP port of Shiki. Syntax-highlight code with VS Code themes and TextMate grammars — no extensions, no Node, no Oniguruma binding.

Go to Download

Rudl UDP cluster logging library (PSR4 compliant)

Go to Download

The phplrt component for assembling a lexer

Go to Download

Dependency-free full-text search for PHP. BM25 ranking, fuzzy and boolean modes, search-as-you-type prefix matching, stopword filtering and Snowball stemming for 62 languages, snippet extraction and result highlighting — one SQLite file, zero infrastructure.

Go to Download

A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.

Go to Download

Flexible, strategy-based text chunking for Laravel with support for character, token, sentence, and markdown-aware splitting. Perfect for AI/LLM applications and RAG systems.

Go to Download

A simple library for make Lexer and Parsers to build a language

Go to Download