Theme clustering

The process of grouping raw customer signals into themes based on shared meaning, root cause, or required fix — the core action of a product intelligence tool.

Theme clustering is what happens between a raw signal stream and an actionable roadmap. The work is straightforward to describe and hard to do well: read every signal, decide which ones are about the same underlying need, group them, name the group. Modern tools do this with embeddings and LLMs; older tools did it with humans and stickies; the underlying decision — same problem or different problem — is the same either way.

The hard call is the split test. Two signals that look superficially similar may require different engineering work; two signals that sound completely different may be the same problem in different language. The honest heuristic, and the one Kiln's clustering prompt is built around: if a single engineering change would resolve both signals, they belong to the same theme. If the fixes would land in different sprints, different engineers, or different parts of the codebase, they're different themes.

A clustering system that ignores the split test produces themes that are either too coarse to act on (an "exports" theme that conflates a CSV format request with a fundamental schema change) or too fine to bother with (twelve separate themes for twelve phrasings of the same complaint). Most off-the-shelf NLP clustering fails on one of those two axes; the good systems are the ones that have the split-test logic written into the prompt.