The cost of being wrong decides the design
In one line
The same model sits behind an autocomplete and a legal notice; what changes is what a wrong answer costs, and that — not the difficulty — sets how much checking goes around it.
The situation
Five features are proposed in the same quarter: autocomplete in the ticket editor, a draft reply for agents, a summary of each support call, automatic approval of small refunds, and a generated legal notice for account termination. The team is asked to estimate them.
Technically they are not very different — each is a prompt, a model call and some text back. The right estimates differ by an order of magnitude anyway, and the reason has nothing to do with the model. It has to do with what happens the day the output is wrong.
Sort by consequence, not by difficulty#
Put the five on a line from "cheap to be wrong" to "ruinous to be wrong".
Autocomplete is at the cheap end. A bad suggestion costs the agent a keystroke to ignore. The model can be wrong a third of the time and the feature still helps. Ship it, measure acceptance rate, move on.
A draft reply is one step along. A wrong draft that gets sent is a bad customer interaction — but an agent reads it first, so the design question is "how do we make sure the agent actually reads it" rather than "how do we make the model perfect". Show the sources, make editing frictionless, log what was changed.
A call summary is further along than it looks. Nobody checks a summary; that is the point of it. If it states the customer agreed to something they did not, that error propagates into the account notes forever. The design needs the summary to quote rather than paraphrase for anything binding, and a here is a data-integrity incident, not a typo.
Automatic refund approval is near the ruinous end. Money moves. The model's output is a decision, so it should come back as with a confidence, and there should be a in code — amount thresholds, rate limits, an audit trail — that applies whatever the model says. Below a threshold it may run alone; above it, a human clicks.
A generated legal notice is at the far end. One wrong clause is a lawsuit. The model drafts, a lawyer signs, and the system is built so that nothing leaves without the signature. Here the model is a productivity tool for the expert, and any design that removes the expert is wrong regardless of quality.
What the line tells you#
Each position on the line implies a different amount of machinery around the same model call. That machinery — the review step, the confidence threshold, the audit log, the human click — is where the estimate goes, and it is why five technically similar features cost such different amounts.
It also tells you where to start. The cheap end ships first and teaches you how the model behaves on your data. The expensive end waits until the evals from the cheap end exist, and even then it launches with a human in the loop that is removed only when the numbers earn it.
Questions to ask
- If this output is wrong, who is hurt and how badly?
- Who sees it before it takes effect — a person, code, or nobody?
- What is the cheapest check that would catch the worst case?
- Which end of the line does this sit at, and is the estimate consistent with that?
Mindset
Estimate by consequence, not by difficulty. The model's capability is roughly constant across features; the machinery you must build around it scales with what a mistake costs. Put the feature on the line first, and the design follows.
Goes deeper in
Phase 3 · 6 min
Structured output
Getting a schema-valid object back instead of prose, so the boundary between the model and your code stops being a parsing problem.
Phase 4 · 7 min
Agents & tool use
A loop where the model picks a tool and your code decides whether to run it, and every interesting problem is in the loop rather than in the model.
Phase 4 · 6 min
Failure modes
The failures that matter here do not raise exceptions — they return 200 with the wrong thing — so the defences are structural rather than reactive.
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.