Libraries tagged by tokenized

componenta/tokenizer

0 Favers
9 Downloads

Token-based PHP source inspection helpers for Componenta

sastrawi/tokenizer

28 Favers
947 Downloads

PHP library that allows you to tokenize Bahasa Indonesia.

kevintweber/html-tokenizer

5 Favers
47765 Downloads

Will tokenize HTML.

devtronic/super-tokenizer

1 Favers
70 Downloads

A powerful dynamic tokenizer written in PHP

yurunsoft/tiktoken

0 Favers
276 Downloads

PHP version of tiktoken

yaroslavpopovic/laravel-text-chunker

0 Favers
27 Downloads

Flexible, strategy-based text chunking for Laravel — fork of droath/laravel-text-chunker with Laravel 13 support.

vanderlee/comprehend

8 Favers
17 Downloads

Framework for building BNF LR(1) parsers

turanjanin/serbian-language-tools

19 Favers
2358 Downloads

Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.

teepluss/wordbreaker

4 Favers
26 Downloads

Word breaker is Thai wording separator for using with fulltext software.

phpwatch/phptoken-polyfill

10 Favers
19 Downloads

A polyfill for PHP 8.0's PhpToken class

patchwork/php-parser

6 Favers
27 Downloads

Fast and extensible PHP code analysis and transformation

onstage2426/fuzor

1 Favers
67 Downloads

Dependency-free full-text search for PHP. BM25 ranking, fuzzy and boolean modes, search-as-you-type prefix matching, stopword filtering and Snowball stemming for 62 languages, snippet extraction and result highlighting — one SQLite file, zero infrastructure.

jawitold/fqcn-extractor

0 Favers
61 Downloads

A lightweight PHP utility to extract the Fully Qualified Class Name (FQCN) from a PHP source file using the built-in tokenizer.

hosmelq/search-syntax-parser

8 Favers
785 Downloads

Parse search queries into structured data with field searches, boolean logic, ranges, lists, and adapters.

edin/lexicon

4 Favers
28 Downloads

Attribute-driven PHP lexer and parser toolkit with enum tokens, grammar recipes, diagnostics, and AST/BNF debugging.

<< Previous Next >>