Libraries tagged by tokeniser
rahul900day/gpt-3-encoder
32443 Downloads
GPT-3-Encoder is a PHP implementation of OpenAI's original python encoder/decoder.
padosoft/laravel-pii-redactor
472 Downloads
Italian-first PII redaction pipeline for Laravel: deterministic regex + checksum-validated detectors (codice fiscale, P.IVA, IBAN), pluggable strategies (mask / hash / tokenise / drop), GDPR + EU AI Act ready.
textualization/ropherta-tokenizer
627 Downloads
GPT3Tokenizer (BPE) with Roberta-base vocabulary.
ride/lib-tokenizer
8074 Downloads
Tokenizer library of the Ride framework.
ricwein/tokenizer
1553 Downloads
configurable string tokenizer
ltd-beget/sphinx-configuration-tokenizer
23971 Downloads
tokenize sphinx configuration and that's all, folks.
kerasai/behat-tokenizer
21138 Downloads
Token replacement for Behat test suite.
codewithkyrian/tokenizers
35 Downloads
Fast, pure-PHP tokenizer library compatible with Hugging Face tokenizers for encoding and decoding text
yannickl88/css-tokenizer
30945 Downloads
CSS tokenizer
hippophp/tokenizer
1311 Downloads
Wrapper around token_get_all
turanjanin/serbian-language-tools
2358 Downloads
Set of tools for tokenization, transliteration and diacritic restoration of a text written in Serbian language.
tacoberu/bnf
3257 Downloads
Framework for building BNF like parsers.
popsul/php-token-reflection
78495 Downloads
Library emulating the PHP internal reflection using just the tokenized source code.
hybridlogic/classifier
1445 Downloads
A Naive Bayesian classification library for PHP with support for different tokenizers to optimize string classification.
droath/laravel-text-chunker
271 Downloads
Flexible, strategy-based text chunking for Laravel with support for character, token, sentence, and markdown-aware splitting. Perfect for AI/LLM applications and RAG systems.