Two different JSON stories

Chunk JSON in. Write JSON out.

json_chunker splits a JSON or JSONL file on objects, arrays, keys, size, or depth. Separately, every strategy can dump its chunks as JSON for a pipeline. Those are not the same file format.

1. Output JSON (any strategy)

This is the connector into RAG, a vector DB, or your own job. CLI:

python -m chunking_strategy chunk doc.txt --strategy sentence_based --format json -o chunks.json
python -m chunking_strategy chunk data.json --strategy json_chunker --format json -o out.json

The file looks like this:

{
  "metadata": {
    "strategy_used": "sentence_based",
    "total_chunks": 12,
    "processing_time": 0.02,
    "source_info": {"source": "doc.txt"}
  },
  "chunks": [
    {
      "id": "...",
      "content": "The text of this chunk...",
      "modality": "text",
      "metadata": {
        "source": "doc.txt",
        "chunker_used": "sentence_based"
      }
    }
  ]
}

CLI --format json may shorten a very long content field for display safety (about 1000 characters). If you need the full text plus offsets, serialize in Python with chunk.to_dict().

import json
from chunking_strategy import create_chunker

result = create_chunker("sentence_based").chunk(open("doc.txt").read())
payload = {
    "strategy_used": result.strategy_used,
    "quality_score": result.quality_score,
    "chunks": [c.to_dict() for c in result.chunks],
}
json.dump(payload, open("chunks.json", "w"), indent=2)

Each to_dict() chunk includes:

Field	Meaning
`id`	Stable id for this chunk.
`content`	Full text (or bytes for media).
`start` / `end`	Inclusive / exclusive offsets. Unit is Unicode scalars unless metadata says otherwise.
`parent_id` / `children_ids`	Hierarchical `recursive` tree.
`token_count`	Set when a real tokenizer ran, not chars/4.
`metadata`	`source`, `chunker_used`, plus extras such as `json_start_index`, `symbol_name`, CSV headers.

Other CLI dumps: --format yaml, --format text, --summary-only (no file), --quality-report.

2. json_chunker (splitting JSON / JSONL)

Use this when the input is JSON and you do not want a raw character window to cut through an object.

from chunking_strategy import create_chunker

# Array of objects: N objects per chunk (default 100)
create_chunker("json_chunker", chunk_by="objects", objects_per_chunk=25)

# One array, N elements per chunk
create_chunker("json_chunker", chunk_by="array_elements", elements_per_chunk=200)

# Group records that share a key
create_chunker("json_chunker", chunk_by="key_groups", group_by_key="category")

# Size budget
create_chunker("json_chunker", chunk_by="size_limit", size_limit_mb=5)

# Stop descending nested objects past a depth
create_chunker("json_chunker", chunk_by="depth_level", max_depth=2)

python -m chunking_strategy chunk events.jsonl --strategy json_chunker -o events_chunks.json
python -m chunking_strategy chunk catalog.json --strategy json_chunker --format json

preserve_structure defaults to true so each chunk is still valid JSON. JSONL / NDJSON is a supported input. Metadata extras include json_start_index, json_end_index, json_object_count. The lab runs the same name; Python re-dumps JSON, so byte-for-byte strings can differ from the tab.

Streaming JSONL on a large corpus: config_examples/use_case_configs/jsonl_streaming.yaml and Config and scale.