pip install chunking-strategy[ml]

Cut on meaning, then embed

Two ML jobs. semantic decides where to cut using sentence embeddings. embed takes already-cut chunks and writes vectors for a database. The lab only does a Transformers.js MiniLM preview of the first job. It will not match pip.

Install

pip install chunking-strategy[ml]
pip install chunking-strategy[tiktoken]   # if you also want cl100k windows

Missing extra raises MissingExtraError with that pip line. Default embedding model is all-MiniLM-L6-v2. [text] no longer pulls torch; that was a 0.5.0 change.

semantic: topic boundaries

Sentences are embedded, then a new chunk starts when similarity to the previous context drops below a threshold (default 0.7). Lower threshold = fewer, longer chunks. Higher = more cuts.

from chunking_strategy import create_chunker

c = create_chunker(
    "semantic",
    embedding_model="all-MiniLM-L6-v2",
    similarity_threshold=0.7,
    min_chunk_sentences=3,
    max_chunk_sentences=15,
    boundary_detection="similarity_threshold",  # or sliding_window, dynamic_threshold, coherence_based
)
result = c.chunk(open("article.txt").read())

python -m chunking_strategy chunk article.txt --strategy semantic --format json -o semantic.json

Aliases: semantic_chunker, semantic_chunking. Related pip-only names: embedding_based, context_enriched, discourse_aware, ml_cdc (grey in the tab until a JS port exists).

embed: vectors after the cut

python -m chunking_strategy list-models
python -m chunking_strategy embed doc.txt --strategy sentence_based --model all-MiniLM-L6-v2 -o vectors.json
python -m chunking_strategy embed-batch ./corpus --strategy recursive_character -o ./vectors

Each exported row is built for a vector DB: id, vector (dimension of the model), payload (text + metadata). Destinations people actually use: Qdrant, Weaviate, Pinecone, Chroma. Flags: --batch-size, --normalize / --no-normalize, --device cuda|cpu, --export-format json|dict.

from chunking_strategy import create_chunker
from chunking_strategy.core.embeddings import (
    EmbeddingModel, EmbeddingConfig, embed_chunking_result, export_for_vector_db,
)

chunks = create_chunker("sentence_based").chunk(open("doc.txt").read())
cfg = EmbeddingConfig(model=EmbeddingModel("all-MiniLM-L6-v2"), batch_size=32)
embedded = embed_chunking_result(chunks, cfg)
print(export_for_vector_db(embedded, format="json")[:500])

Worked examples: examples/03_embedding_workflows.py, 05_embeddings_integration_demo.py, 16_verify_embedding_setup.py, 18_langchain_integration_demo.py.

YAML when ML is the point

config_examples/embedding_optimized.yaml and rag_system.yaml keep chunk sizes even so embedding batches stay regular. multimodal_embeddings.yaml is the text+image profile. Load with --config on chunk, batch, or embed. See Config and scale.