chunking-strategy 0.5.0

Two surfaces. One set of names.

The playground is a tab that compares two chunkers on a file you never upload. The library is the same names on disk: N-way compare, a chunker you wrote, streaming, layout PDFs, audio, video. This site is the map. The repo READMEs stay as the long form; you should not need them to get moving.

The lab

Drop a file. Two columns. Click a chunk, see the span. Cap 150 MB. Nothing leaves the browser.

CLI only

Install, chunk to JSON, batch a directory, compare, embed. No Python file required.

JSON

The file a pipeline reads, and json_chunker for JSON/JSONL inputs.

ML and embeddings

semantic decides the cut. embed writes vectors. Lab MiniLM is a preview.

Config and scale

YAML profiles, workers, streaming. How a corpus actually runs.

Logs and debug

JSON lines for a log stack. A zip of JSON files for a bug report.

pip install chunking-strategy

Same strategy names, then directories, extras, and no 150 MB cap.

What the tab will not do

Colleagues often ask for two things that already exist in Python and will not be faked in JavaScript:

N-way compare

The lab is A vs B. Three or more strategies is chunking-strategy compare FILE -s a,b,c.

Bring your own strategy

Subclass BaseChunker, @register_chunker, then compare / benchmark against the builtins. The tab will not eval your JS.

Five minutes

Open the lab. Starter text is already fixed_size vs sentence_based. Click a chunk. Drag the divider under the editor if the paste box is too short, or the sidebar divider if the lists feel tight.

pip install chunking-strategy then:

from chunking_strategy import create_chunker
result = create_chunker("sentence_based", max_sentences=2).chunk(open("doc.txt").read())
print(len(result.chunks), result.quality_score)

Need three names on one file?

chunking-strategy compare doc.txt -s fixed_size,sentence_based,recursive_character

Need a cut this registry does not have? Copy examples/custom_algorithms/ and register it. Details in Bring your own.

How the two surfaces sit

file ─┬─► lab tab ── A | B ── your eyes │ (22 live names, 150 MB, PDF/Word extract) │ └─► pip chunking-strategy create_chunker / CLI / orchestrator N-way compare · custom · stream · media 40+ names, extras, no tab cap

The contract is browser ⊆ python. Every live lab name is a real pip name. Grey names in the dropdowns ship in the library and stay grey until a real JS port exists. The lab will not invent token counts as chars/4, and it will not run semantic until MiniLM actually loads.

Chapters

Page	When you open it
The lab	Drop, extract, knobs, share URL (never the file), Find, sidebar width, source height, privacy.
Install	Core vs extras. Skip `[all]` unless you need torch, OpenCV, and Tika together.
CLI only	Install, chunk to JSON, batch a directory, compare, embed. No Python file.
JSON	Output JSON for pipelines, and `json_chunker` for JSON/JSONL inputs.
ML and embeddings	`semantic` cuts, `embed` vectors, lab MiniLM vs pip `[ml]`.
Config and scale	YAML profiles, workers, streaming, directories.
Logs and debug	JSON log lines, `debug collect`, what is in the zip.
Strategies	All 43 registry names, lab / grey / pip-only. `recursive` vs `recursive_character`.
N-way compare	`compare` table vs `benchmark` timings, including custom files.
Bring your own	`@register_chunker`, CLI `custom`, worked examples.
Python API	`create_chunker`, orchestrator, pipeline, adaptive, metrics, streaming.
Formats	What the tab extracts vs what `pdf_chunker` / Tika / media do on disk.

Still only in the repo

Checked against the library public API and CLI. These pages now cover every command people actually run and every name in chunking_strategy.__all__ that is a getting-started surface. Left as runnable files, not extra HTML:

LangChain: examples/18_langchain_integration_demo.py
Streamlit UI: examples/19_streamlit_app_demo.py
Per-knob audio / video / image parameters: strategy modules under chunking_strategy/strategies/multimedia/
Detectors (encoding, language, file type): used by the orchestrator; not a separate product
token_packing, multi_strategy, production_benchmark, enhanced_orchestrator: internals / extras, not first-run

CLI_REFERENCE.md lists hardware --json and debug system-info. Those flags are not in cli.py. Trust these pages and --help.

The long Markdown in the repo (API_REFERENCE.md, DEBUGGING_GUIDE.md, CUSTOM_ALGORITHMS_GUIDE.md, examples/01_*.py) is still there. These pages are the path you read first.