No Python file required

CLI only: install, chunk, ship JSON

If you do not want an API session, this is the whole path. Copy any block with the button on the snippet.

0. Install and prove it

pip install chunking-strategy
python -m chunking_strategy --version
python -m chunking_strategy --help
python -m chunking_strategy list-strategies --tier lab
python -m chunking_strategy list-strategies --tier all
python -m chunking_strategy list-strategies --format json
python -m chunking_strategy list-strategies --format simple

Global flags on every command: -v / -q, --debug, --log-level silent|minimal|normal|verbose|debug|trace, --log-file PATH. --log-file is human text. JSON lines are Python: Logs and debug.

1. One file → JSON

python -m chunking_strategy chunk notes.md
python -m chunking_strategy chunk notes.md --strategy markdown_chunker --format json -o chunks.json
python -m chunking_strategy chunk notes.md -s sentence_based --max-sentences 3
python -m chunking_strategy chunk notes.md --validate --quality-report
python -m chunking_strategy chunk huge.txt --summary-only
python -m chunking_strategy test-strategy sentence_based
python -m chunking_strategy chunk report.pdf --strategy pdf_chunker --format json -o chunks.json

--format is json (default), text, or yaml. How that JSON is shaped: JSON. --quality-report reads the file as UTF-8 text, so use it on markdown/txt, not on a PDF. pdf_chunker needs pip install chunking-strategy[pdf].

2. A directory

python -m chunking_strategy batch *.txt --strategy sentence_based --workers 4 --show-progress
python -m chunking_strategy batch *.py --strategy python_code -o ./chunks --format json
python -m chunking_strategy batch-directory ./corpus --strategy auto
python -m chunking_strategy process-directory ./corpus --config config_examples/enhanced_auto_strategy.yaml
python -m chunking_strategy batch *.pdf --parallel-mode process --workers 4 --validate --quality-report

--parallel-mode is auto | sequential | thread | process. Scale and YAML: Config and scale.

3. Pick a name without the lab

python -m chunking_strategy compare doc.txt -s fixed_size,sentence_based,recursive_character
python -m chunking_strategy compare doc.txt -s json_chunker,fixed_size --format json
python -m chunking_strategy benchmark doc.txt --quick
python -m chunking_strategy benchmark doc.txt --strategies "fixed_size,sentence_based" --runs 3
python -m chunking_strategy benchmark doc.txt --custom-algorithms ./my_chunker.py --output-dir ./bench

Tables and what quality_score means: N-way compare.

4. Embeddings, still CLI

pip install 'chunking-strategy[ml]'
python -m chunking_strategy list-models
python -m chunking_strategy embed doc.txt --strategy sentence_based --model all-MiniLM-L6-v2 -o vectors.json
python -m chunking_strategy embed-batch ./corpus --strategy recursive_character -o ./vectors

Models, thresholds, and lab vs pip: ML and embeddings.

5. Your own chunker, still CLI

python -m chunking_strategy custom load ./my_chunker.py
python -m chunking_strategy custom load-dir ./examples/custom_algorithms
python -m chunking_strategy custom list --detailed
python -m chunking_strategy custom create-template ./my_chunker.py --algorithm-name my_word_count
python -m chunking_strategy custom validate ./my_chunker.py
python -m chunking_strategy custom validate-batch ./examples/custom_algorithms
python -m chunking_strategy custom validate-config ./my-config.yaml
python -m chunking_strategy custom benchmark ./my_chunker.py --compare-with sentence_based --compare-with fixed_size

How to write the file: Bring your own.

6. Logs, debug zip, hardware

python -m chunking_strategy init-config
python -m chunking_strategy --log-file chunking.log chunk notes.md
python -m chunking_strategy --debug chunk the-bad-file.pdf
python -m chunking_strategy debug enable --log-file my-debug.log
python -m chunking_strategy debug collect -d "PDF extract throws" -o ./debug-archives/
python -m chunking_strategy debug archive "PDF extract throws"
python -m chunking_strategy debug test-logging
python -m chunking_strategy hardware --detailed --recommendations

JSON log lines and what is inside the zip: Logs and debug. There is no hardware --json and no debug system-info in the shipped CLI; CLI_REFERENCE.md is wrong on those two. Hardware JSON is system_info.json in the debug zip, or get_hardware_info() in Python.

Recipes in the repo: examples/04_cli_examples.py.