Is this an AI problem at all?
In one line
Three questions to ask before naming a model, because most requests that arrive as "add AI" resolve on the first one.
The situation
A product manager drops a ticket on the board: "Add AI to the support inbox." No further detail. The team's instinct is to open a chat API and start prototyping, because that is the fastest way to have something to show.
Six weeks later there is a demo that answers questions plausibly, nobody can say whether it is better than what was there before, and the only metric anyone can point to is "it feels smart". This is the most common shape of a first AI project, and it is avoidable in an afternoon — before any model is chosen — by asking what kind of problem this actually is.
Three questions, in order#
Is there a deterministic rule? A surprising share of "AI" requests are routing rules that nobody wrote down. If tickets containing the word "invoice" should go to billing, that is a string match, and a string match is faster, cheaper, testable and never . Only when the rule genuinely cannot be written — when a human reading the ticket makes a judgement call that they could not fully explain — is a model the right tool.
Is the output a judgement on messy text, or an exact fact? Models are good at the first and unreliable at the second. "Is this customer angry" is a judgement. "What is this customer's current balance" is a fact, and it should come from the billing database, with the model at most phrasing the sentence around it. Confusing the two is how a system ends up confidently inventing an account balance.
What happens when it is wrong? Not if — when. Every model output is sometimes wrong, so the question is what a wrong one costs and who catches it. An internal draft that a human edits before sending tolerates a lot. An automated refund tolerates nothing, and needs a check in code or a person in the loop regardless of how good the model is.
What the ticket becomes#
Run the support inbox through those three questions and it usually decomposes into four different things. Routing tickets to the right queue: a rule where one exists, a small classifier where one does not. Suggesting an answer from the help centre: a judgement on messy text, good fit, with the source shown so the agent can check it. Looking up the customer's order status: a fact, from the database, never from the model. Sending the reply: a human clicks.
That is a plan with a shape, a way to measure each piece, and a clear line around what the model is trusted to do. The for each part can be written before the first prototype, because the parts now have definitions.
Questions to ask
- Could I write this as a rule, even a long one? If yes, write the rule.
- Is the answer a judgement or a fact? Facts come from systems, not models.
- Who sees a wrong output first — a customer, or someone who can catch it?
- What would "better than today" mean as a number I could measure?
- Which single piece of this, done alone, would already be useful?
Mindset
"Add AI" is not a requirement; it is a request to be decomposed. The skill is not knowing what a model can do — it is noticing that most of the ticket was never a model problem, and that the part that is has a cost-of-error attached to it that the design has to answer for.
Goes deeper in
Phase 3 · 6 min
Structured output
Getting a schema-valid object back instead of prose, so the boundary between the model and your code stops being a parsing problem.
Phase 3 · 6 min
Evals
Automated tests for a component whose output is not stable, where the assertion is a threshold on a score and the suite is the only thing making changes measurable.
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.