Libraries tagged by tokenized

tokenize dns zone files and that's all, folks.

Go to Download

Fast, pure-PHP tokenizer library compatible with Hugging Face tokenizers for encoding and decoding text

Go to Download

Function parser for PHP functions, methods, and closures

Go to Download

Unilex: lexical analyzer generator with Unicode support written in PHP

Go to Download

🌳 PhpSyntax: a lossless concrete syntax tree (CST) for PHP. Every token, space and comment is in the tree and printing it gives the file back byte for byte, so a tool changes only what it touches. Comes with a mutation API for refactoring, formatters and code generation.

Go to Download

A set of phplrt lexer abstractions

Go to Download

The phplrt lexer library that can be used for scanners implementation using PCRE

Go to Download

A token-based PHP source code obfuscator with full support for PHP 8.3 through 8.5.

Go to Download

Flexible, strategy-based text chunking for Laravel with support for character, token, sentence, and markdown-aware splitting. Perfect for AI/LLM applications and RAG systems.

Go to Download

A tokenizer for Doctrine annotations.

Go to Download

Text tokenizing

Go to Download

PHP Text Tokenizer for GPT models

Go to Download

Tokenizer for PHP code

Go to Download

High-Performance Tokenizer implementation in PHP

Go to Download

Security Tokens

Go to Download