Library, not the tab
Your strategy is Python. The tab will not eval it.
If the cut you want is not in the dropdown, do not paste a generated worker into the playground. Register a BaseChunker. Then create_chunker, the CLI, compare, and benchmark all see it. If it belongs in the lab later, port it with a golden fixture. That order is the point.
Minimum chunker
from chunking_strategy.core.base import (
BaseChunker, Chunk, ChunkingResult, ChunkMetadata, ModalityType,
)
from chunking_strategy.core.registry import register_chunker
@register_chunker(
name="my_word_count_chunker",
category="text",
description="Chunks on a word budget",
default_parameters={"words_per_chunk": 50},
)
class MyWordCountChunker(BaseChunker):
def __init__(self, words_per_chunk=50, **kwargs):
super().__init__(name="my_word_count_chunker", category="text", **kwargs)
self.words_per_chunk = words_per_chunk
def chunk(self, content, source_info=None, **kwargs):
if isinstance(content, bytes):
content = content.decode("utf-8")
words = str(content).split()
chunks = []
for i in range(0, len(words), self.words_per_chunk):
text = " ".join(words[i:i + self.words_per_chunk])
chunks.append(Chunk(
id=f"word_{i // self.words_per_chunk}",
content=text,
modality=ModalityType.TEXT,
metadata=ChunkMetadata(
source=(source_info or {}).get("source", "unknown"),
chunker_used="my_word_count_chunker",
extra={"word_count": len(text.split())},
),
))
return ChunkingResult(chunks=chunks)
from chunking_strategy import create_chunker
print(create_chunker("my_word_count_chunker", words_per_chunk=40).chunk(text).chunks)
CLI
chunking-strategy custom load ./my_chunker.py
chunking-strategy custom load-dir ./examples/custom_algorithms
chunking-strategy custom list --detailed
chunking-strategy custom create-template ./my_chunker.py --algorithm-name my_word_count
chunking-strategy custom validate ./my_chunker.py
chunking-strategy custom validate-batch ./examples/custom_algorithms
chunking-strategy custom validate-config ./my-config.yaml
chunking-strategy custom benchmark ./my_chunker.py --compare-with fixed_size --compare-with sentence_based
chunking-strategy benchmark doc.txt --custom-algorithms ./my_chunker.py
from chunking_strategy import load_custom_algorithm, create_chunker
load_custom_algorithm("./my_chunker.py")
print(create_chunker("my_word_count_chunker").chunk(text).chunks)
Worked examples in the repo
| File | What it shows |
|---|---|
examples/custom_algorithms/balanced_length_chunker.py | Target length with sentence/word tolerance. |
examples/custom_algorithms/regex_pattern_chunker.py | Ordered regex splits with a size fallback. |
examples/custom_algorithms/sentiment_based_chunker.py | Domain signal (sentiment shifts) as boundaries. |
examples/08_extensibility_demo.py | Register, preprocess, plug into the orchestrator. |
CUSTOM_ALGORITHMS_GUIDE.md is the long form (validation, YAML, parallel custom runs). Start with the three files above.
Getting it into the lab later
- Python chunker + tests.
- A golden under
fixtures/so JS and Python agree on offsets (Unicode scalar offsets, not UTF-16). - A JS port in
poc/chunkers/, then move the name fromlab_later/python_onlytolabinschemas/strategy-tiers.yaml.
The tab will not run pasted JS, and it will not
eval a custom algorithm. That is how you keep browser ⊆ python and keep other people's files off a script you did not ship.