Designing for when the model is wrong

When people talk about building with machine learning, the conversation gravitates to accuracy: benchmark numbers, model sizes, evaluation scores. Those matter. But after building a few AI-shaped products, I have come to think they are the least interesting part of the work. The experience a person actually has is decided somewhere else — in the handful of moments when the model is wrong, and the interface has to decide what happens next.

The tail is the product

A classifier that is 92% accurate sounds excellent until you consider the 8%. If those failures are distributed across a user's most important tasks, the felt reliability is far lower than the number suggests. People do not average their experience; they remember the time the tool confidently mislabelled something that mattered.

This is why I have stopped treating the error rate as a single figure to minimise and started treating the error experience as a surface to design. Two products with identical accuracy can feel completely different depending on how they handle being wrong.

A model's accuracy is a property of the model. A product's trustworthiness is a property of its interface.

Three moves that help

1. Show confidence, and mean it

If the system has a notion of how sure it is, the interface should too. Surfacing confidence is not about displaying a probability — most people do not want a number — it is about ordering the experience so that uncertain results ask for attention and confident ones recede. In a categorisation tool I built, the single most effective change was simply sorting low-confidence rows to the top.

2. Make correction a first-class action

When the model is wrong, fixing it should feel like teaching, not like fighting an error state. That means correction lives in the main flow, is instant, and — ideally — visibly improves future behaviour. A tool that learns from being corrected earns a kind of patience that a static one never will.

• Corrections belong in the primary interface, not a settings screen.
• Every correction should feel like it changed something.
• Never punish the user for the model's mistake with friction.

3. Let the system abstain

The most underrated behaviour in an AI product is the honest 'I'm not sure'. A retrieval system I worked on became dramatically more trusted the moment it started refusing to answer when its evidence was thin. Abstention feels like a loss on a benchmark and a win in practice.

function answer(query: Query): Answer {
  const evidence = retrieve(query);
  // Abstaining is a valid, designed outcome — not a failure path.
  if (evidence.score < THRESHOLD) {
    return { kind: "abstain", reason: "insufficient evidence" };
  }
  return generate(query, evidence);
}

None of this is exotic. It is ordinary product craft applied to a probabilistic core. And that, I think, is the point: building with AI is less a new discipline than an old one — designing for uncertainty — with the uncertainty moved into the centre of the product.

Where this leaves me

I still care about model quality. But when I sit down to build something now, I start from the failure cases and work backwards. What does the product do when it does not know? What does correction feel like? Where does it earn the right to be trusted? Answer those well, and a modest model can carry a genuinely good product. Answer them poorly, and no benchmark will save you.