Library

Python is the pipeline. The tab is the preview.

Most calls are create_chunker(name, **kwargs).chunk(text). The orchestrator is for files, auto-pick, batch, and streaming. Config YAML is for repeating that choice across a directory. Full signatures live in API_REFERENCE.md; this page is the path you actually type.

create_chunker

from chunking_strategy import create_chunker, list_chunkers

list_chunkers(category="text")          # also: list_strategies
c = create_chunker("markdown_chunker", chunk_by="headers", header_level=2)
result = c.chunk(open("README.md").read())
for ch in result.chunks:
    print(ch.id, ch.content[:80], ch.metadata)

result.quality_score is 0.4 size + 0.4 sentence-boundary + 0.2 span coverage. CDC / code / JSON skip the sentence term.

Orchestrator

from chunking_strategy import ChunkerOrchestrator

orch = ChunkerOrchestrator(config={"strategies": {"primary": "auto"}})
r = orch.chunk_file("script.py")        # auto: paragraph for code
r = orch.chunk_file("notes.txt", strategy_override="sentence_based")
orch.chunk_files_batch(["a.txt", "b.md"], workers=4)

Auto-pick (file extension + size) is a starting point, not magic: code → paragraph, text → sentence, data → fixed_size, large files → rolling_hash. Override when you already know.

Any strategy on any file

from chunking_strategy import apply_universal_strategy
apply_universal_strategy("sentence", "document.pdf")
apply_universal_strategy("fixed_size", "script.py", chunk_size=1000)

Universal extract-then-chunk is how you keep one window size across a mixed corpus. Specialized chunkers (python_code, pdf_chunker) still win when you care about structure.

Streaming and mmap

The tab caps at 150 MB because the file is in RAM. On disk, StreamingChunker yields as it reads; MemoryMappedStreamer uses mmap for huge files. Orchestrator switches to a streaming path for large inputs. See examples/06_streaming_benefits_demo.py and 10_enhanced_streaming_demo.py.

from chunking_strategy.core.streaming import StreamingChunker
for chunk in StreamingChunker("sentence_based").stream_file("huge.txt"):
    index(chunk)

Embeddings

python -m chunking_strategy embed doc.txt --strategy sentence_based
python -m chunking_strategy list-models

[ml] pulls sentence-transformers. The lab semantic path is Transformers.js MiniLM in the tab; those cuts do not match pip. Full path: ML and embeddings.

Config YAML

Eight profiles under config_examples/. Start with enhanced_auto_strategy.yaml. Corpus walk, workers, and streaming: Config and scale. JSON in and out: JSON.

python -m chunking_strategy init-config
python -m chunking_strategy chunk report.pdf --config config_examples/document_first_config.yaml

Pipeline (chunk, then filter, then merge)

from chunking_strategy import ChunkingPipeline
from chunking_strategy.core.pipeline import ChunkerStep, FilterStep, MergeStep

pipeline = ChunkingPipeline([
    ("chunker", ChunkerStep("sentence_chunker", "sentence_based", max_sentences=2)),
    ("filter", FilterStep("size_filter", min_size=10)),
    ("merge", MergeStep("merge_small", min_chunk_size=80)),
])
result = pipeline.process(open("doc.txt").read())

ChunkingPipeline.from_config({...}) is the JSON import of the same steps. This is a processing chain. It is not the registry strategy named adaptive.

Adaptive wrapper vs adaptive strategy

from chunking_strategy import AdaptiveChunker
from chunking_strategy.core.adaptive import FeedbackType

chunker = AdaptiveChunker("sentence_based")
result = chunker.chunk(open("doc.txt").read())
result = chunker.chunk_with_feedback(open("doc.txt").read(), 0.3, FeedbackType.QUALITY)

AdaptiveChunker wraps another strategy and retunes it from quality/latency feedback. create_chunker("adaptive") is a separate python-only strategy. Demo: examples/22_adaptive_chunking_learning_demo.py.

Quality metrics

from chunking_strategy import create_chunker
from chunking_strategy.core.metrics import ChunkingQualityEvaluator

result = create_chunker("sentence_based").chunk(open("doc.txt").read())
metrics = ChunkingQualityEvaluator().evaluate(result, open("doc.txt").read())
print(metrics.overall_score, metrics.size_consistency, metrics.coherence)

CLI: chunk notes.md --validate --quality-report. That report reads the file as UTF-8 text. Scores live on 0–1. Same evaluator as examples/07_metrics_collection_demo.py and 21_comprehensive_metrics_demo.py.

JSON logs

configure_logging(..., format_json=True, log_file="chunking_metrics.jsonl") then debug collect for a zip. Full path: Logs and debug.

Examples in order

01_basic_usage.py through 22_adaptive_chunking_learning_demo.py. Beginner 01–04, integration 05–08, performance 09–13, production 14–16, then embeddings, Streamlit, LangChain, metrics. Run them from the repo root after pip install -e .. Streamlit is examples/19_streamlit_app_demo.py. LangChain is 18_langchain_integration_demo.py. Those two stay as runnable demos, not extra HTML pages.