Libraries tagged by tokenize

ricwein/tokenizer

1 Favers
1615 Downloads

configurable string tokenizer

funivan/php-tokenizer

12 Favers
38456 Downloads

Wrapper around token_get_all. Easy to extract and modify php code.

everstu/gpt3-tokenizer

0 Favers
334 Downloads

PHP package for Byte Pair Encoding (BPE) used by GPT-3.

sastrawi/tokenizer

28 Favers
947 Downloads

PHP library that allows you to tokenize Bahasa Indonesia.

kevintweber/html-tokenizer

5 Favers
47765 Downloads

Will tokenize HTML.

yaroslavpopovic/laravel-text-chunker

0 Favers
88 Downloads

Flexible, strategy-based text chunking for Laravel — fork of droath/laravel-text-chunker with Laravel 13 support.

vanderlee/comprehend

8 Favers
17 Downloads

Framework for building BNF LR(1) parsers

teepluss/wordbreaker

4 Favers
26 Downloads

Word breaker is Thai wording separator for using with fulltext software.

phpwatch/phptoken-polyfill

9 Favers
19 Downloads

A polyfill for PHP 8.0's PhpToken class

patchwork/php-parser

6 Favers
27 Downloads

Fast and extensible PHP code analysis and transformation

onstage2426/fuzor

1 Favers
128 Downloads

Dependency-free full-text search for PHP. BM25 ranking, fuzzy and boolean modes, search-as-you-type prefix matching, stopword filtering and Snowball stemming for 62 languages, snippet extraction and result highlighting — one SQLite file, zero infrastructure.

kamthorn/thai-break

0 Favers
9 Downloads

Fast, high-accuracy Thai word segmentation & typographic line breaker for PHP and Laravel (dompdf, mPDF, Web)

jawitold/fqcn-extractor

0 Favers
99 Downloads

A lightweight PHP utility to extract the Fully Qualified Class Name (FQCN) from a PHP source file using the built-in tokenizer.

edin/lexicon

6 Favers
28 Downloads

Attribute-driven PHP lexer and parser toolkit with enum tokens, grammar recipes, diagnostics, and AST/BNF debugging.

crazzy501/gpt-3-encoder-php

1 Favers
449 Downloads

PHP BPE Text Encoder/Decoder for GPT-2 / GPT-3. Based on coderevolutionplugins/gpt-3-encoder-php

<< Previous Next >>