Directories, not one tab

One YAML, a thousand files

The lab is how you pick a name. A corpus is how you keep that name. YAML selects a primary strategy per extension, a fallback chain, chunk sizes, and whether to stream. The CLI then walks a directory with workers.

Start from a profile

python -m chunking_strategy init-config
python -m chunking_strategy process-directory ./corpus --config config_examples/enhanced_auto_strategy.yaml
python -m chunking_strategy batch-directory ./corpus --strategy auto --workers 4 --show-progress
python -m chunking_strategy batch ./corpus/*.pdf --config my.yaml --parallel-mode process --workers 4 -o ./chunks

Profile	When
`enhanced_auto_strategy.yaml`	Mixed corpus. Auto primary + fallbacks.
`code_focused_config.yaml`	Repos. `python_code` / `universal_code`.
`document_first_config.yaml`	Prose, PDFs, research.
`large_files_streaming.yaml`	100 MB+ files. Streaming buffers, `fixed_size` primary.
`embedding_optimized.yaml`	Even sizes for a vector index.
`rag_system.yaml`	RAG default sizes.
`multimodal_embeddings.yaml`	Text plus images.
`basic_example.yaml`	Learning. Everything sentence-based.

They live under config_examples/ and config_examples/use_case_configs/ (JSONL streaming, Tika, hardware). CONFIGURATION_GUIDE.md is the long form.

What a profile actually says

profile_name: "large_files_streaming"
strategies:
  primary: "fixed_size"
  fallbacks: ["sentence_based", "paragraph_based"]
  configs:
    fixed_size:
      chunk_size: 2048
      overlap_size: 256
      unit: "character"
strategy_selection:
  ".txt":
    primary: "sentence_based"
    fallbacks: ["paragraph_based", "fixed_size"]
  ".py":
    primary: "python_code"
    fallbacks: ["paragraph_based", "fixed_size"]

If the specialist throws (bad Python syntax, missing extra), the next fallback runs. A CLI -s / strategy_override always wins over the YAML primary.

Streaming instead of RAM

The tab caps at 150 MB because the file is in the browser. On disk:

from chunking_strategy import create_chunker
from chunking_strategy.core.streaming import StreamingChunker, MemoryMappedStreamer

for chunk in StreamingChunker("sentence_based").stream_file("huge.txt"):
    index(chunk)

for chunk in MemoryMappedStreamer(create_chunker("fixed_size")).stream_file("huge.bin"):
    index(chunk)

Orchestrator chunk_file already switches to a streaming path on large inputs. Checkpoints can resume a crashed walk. Demos: examples/06_streaming_benefits_demo.py, 10_enhanced_streaming_demo.py, 11_streaming_and_tika_demo.py, 12_parallelization_demo.py, 13_smart_parallelization_demo.py, 15_comprehensive_directory_processing_demo.py.

Workers

python -m chunking_strategy batch *.txt --workers 4 --parallel-mode auto
python -m chunking_strategy batch *.txt --parallel-mode thread --workers 8
python -m chunking_strategy batch *.pdf --parallel-mode process --workers 4
python -m chunking_strategy hardware

auto picks thread vs process from file count and size. hardware prints what the machine can actually run. POSIX extras [media], [tika], [hardware] stay off Windows CI on purpose.

Custom algorithms in the same run

python -m chunking_strategy benchmark ./corpus/a.txt --custom-algorithms ./my_chunker.py
python -m chunking_strategy custom load-dir ./team_chunkers --recursive

That is how a team keeps one YAML and still plugs a house strategy. See Bring your own.