Libraries tagged by tokenizer

Pure PHP port of Shiki. Syntax-highlight code with VS Code themes and TextMate grammars — no extensions, no Node, no Oniguruma binding.

Go to Download

Rudl UDP cluster logging library (PSR4 compliant)

Go to Download

Library emulating the PHP internal reflection using just the tokenized source code.

Go to Download

Dependency-free full-text search for PHP. BM25 ranking, fuzzy and boolean modes, search-as-you-type prefix matching, stopword filtering and Snowball stemming for 62 languages, snippet extraction and result highlighting — one SQLite file, zero infrastructure.

Go to Download

Rejects rich-text HTML that could run script: runs the HTML5 tokenizer browsers use, reports every problem with the tag or attribute at fault, never rewrites the content

Go to Download

A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.

Go to Download

Token estimation, cost calculation & context-window utilities for LLM-powered Laravel apps

Go to Download

A simple library for make Lexer and Parsers to build a language

Go to Download

This project implements a simple lexer.

Go to Download

Handler-driven tokenizer and AST builder for PHP: define your grammar as small callables, get tokens, a navigable syntax tree, and compilation.

Go to Download

Server-side syntax highlighting for PHP with semantic scopes, embedded languages, and adaptable themes.

Go to Download

GPT-3-Encoder is a PHP implementation of OpenAI's original python encoder/decoder.

Go to Download

Hippo is an open-source tool that helps PHP programmers adhere to certain a given convention.

Go to Download

Custom Bundle Extension (Custom Tokenizers) for Tonics Template System

Go to Download

PHP Library to tokenize various inputs.

Go to Download