Download the PHP package nuglif/chunkytown without Composer
On this page you can find all versions of the php package nuglif/chunkytown. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
FAQ
Example:
In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
- Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
- To use Composer is sometimes complicated. Especially for beginners.
- Composer needs much resources. Sometimes they are not available on a simple webspace.
- If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
- Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Informations about the package chunkytown
ChunkyTown (PHP)
Text chunker for French RAG pipelines. Splits a document into overlapping spans that respect natural boundaries (paragraphs, lists, sentences, words) instead of cutting in the middle of a sentence — and never inside a composed emoji or accented grapheme.
Built on PHP's intl extension (ICU), with a French‑tuned abbreviation
filter so common shortcuts (M. Dupont, St. Patrick, cf., etc.)
are not mistaken for sentence ends.
Installation
Requires PHP 8.1+ with the intl and mbstring extensions.
Quick start
Reusable analysis (multi‑config runs)
If you chunk the same text under several configs (tuning, A/B sweeps),
pre‑compute the boundary detection once with analyze():
Configuration
ChunkerConfig is an immutable value object:
| Option | Default | Meaning |
|---|---|---|
chunkSize |
300 |
Target chunk length in grapheme clusters (user‑perceived characters). |
window |
50 |
± window around the target end to snap to a boundary. |
preferOrder |
['paragraph', 'list', 'sentence', 'word'] |
Boundary kinds tried in order when snapping. |
overlap |
0 |
Desired overlap (graphemes) between successive chunks. |
overlapWindow |
null (falls back to window) |
± window used when locating the start of the next chunk. |
normalize |
false |
Normalize the input to Unicode NFC before analysis (use for mixed NFC/NFD). |
External boundaries
If you already have sentence/paragraph offsets from another parser, merge them in (offsets are in graphemes):
Offsets and slicing
All offsets returned by the chunker are grapheme cluster offsets
(emojis with ZWJ, regional‑indicator flags, skin‑tone modifiers and
NFD‑decomposed accents all count as a single unit). Slice safely with
Chunker::render(), which preserves grapheme integrity for you.
What's special
- ICU sentence detection with a built‑in French abbreviation filter
—
M. Dupont,St. Patrick,cf.,etc.are not mistaken for sentence ends. - ICU word detection — handles apostrophes (
l'arbre), hyphens (peut‑être), and the typical French punctuation cleanly. - Grapheme‑safe: a chunk boundary will never land inside a composed emoji or a decomposed accent.
- Reusable analysis: pay boundary detection once, re‑chunk many times.
Testing
The suite covers the grapheme mapper, every boundary detector,
ChunkerConfig, TextAnalysis, and end‑to‑end chunking — including
parity with the upstream Python implementation on a French fixture.
License
MIT
All versions of chunkytown with dependencies
ext-intl Version *
ext-mbstring Version *