Libraries tagged by tokeniser

devtronic/super-tokenizer

1 Favers
70 Downloads

A powerful dynamic tokenizer written in PHP

yurunsoft/tiktoken

0 Favers
267 Downloads

PHP version of tiktoken

yaroslavpopovic/laravel-text-chunker

0 Favers
27 Downloads

Flexible, strategy-based text chunking for Laravel — fork of droath/laravel-text-chunker with Laravel 13 support.

vanderlee/comprehend

8 Favers
17 Downloads

Framework for building BNF LR(1) parsers

teepluss/wordbreaker

4 Favers
26 Downloads

Word breaker is Thai wording separator for using with fulltext software.

phpwatch/phptoken-polyfill

10 Favers
19 Downloads

A polyfill for PHP 8.0's PhpToken class

patchwork/php-parser

6 Favers
27 Downloads

Fast and extensible PHP code analysis and transformation

onstage2426/fuzor

1 Favers
67 Downloads

Dependency-free full-text search for PHP. BM25 ranking, fuzzy and boolean modes, search-as-you-type prefix matching, stopword filtering and Snowball stemming for 62 languages, snippet extraction and result highlighting — one SQLite file, zero infrastructure.

jawitold/fqcn-extractor

0 Favers
61 Downloads

A lightweight PHP utility to extract the Fully Qualified Class Name (FQCN) from a PHP source file using the built-in tokenizer.

hosmelq/search-syntax-parser

8 Favers
785 Downloads

Parse search queries into structured data with field searches, boolean logic, ranges, lists, and adapters.

edin/lexicon

4 Favers
28 Downloads

Attribute-driven PHP lexer and parser toolkit with enum tokens, grammar recipes, diagnostics, and AST/BNF debugging.

dbeurive/lexer

2 Favers
2761 Downloads

This project implements a simple lexer.

componenta/class-finder

0 Favers
9 Downloads

Lazy class discovery with composable filters and listener notification

chrismcmacken/phptools

25 Favers
3558 Downloads

Small utilities that assist in developing applications

apicart/fql

12 Favers
10714 Downloads

Filter Query Language

<< Previous Next >>