Routing tickets with the wrong kind of model
In one line
A large chat model can sort tickets into queues, at two hundred times the cost of a small classifier that does it better — because the output is a label, not a sentence.
The situation
Support tickets need to land in one of seven queues. The first version does the obvious thing: send each ticket to the chat model with a prompt listing the queues, and parse the reply. It works. It costs about two cents a ticket, takes three seconds, and every so often returns "Billing, I think — or possibly Account" and the parser falls over.
At ten thousand tickets a day that is $200 a day for a decision a human makes in two seconds. The question is not how to prompt better. It is whether this was ever a job for a model that writes.
A label is not a sentence#
The chat model is a : it generates text, one token at a time, and you pay for every token. To get a label out of it you ask for a word, then parse the word. The model is doing something expensive — composing a response — to convey one of seven values.
An does the opposite job. It reads the whole ticket in one pass and produces a representation you can attach a classification head to, giving one score per queue. There is nothing to parse: the output is seven numbers, , and the largest one is the answer. It also gives you something the chat model cannot: a calibrated probability. "Billing, 0.94" is a value your code can act on; "Billing, 0.51 / Account, 0.47" is a signal to send the ticket to a human.
The encoder is around a hundred million parameters, runs in fifteen milliseconds on a CPU, and costs a fraction of a cent. Trained on eight thousand historical tickets — which you already have, because the queues existed before the model did — it typically beats the prompted chat model on accuracy, because it has seen your data and the chat model has seen the internet.
When to reach for which#
The rule that falls out: if the output is a value from a known set, use something built to produce values. If the output is text, use something built to produce text. The chat model with is the reasonable middle when you have no labelled data yet — it gets you a working classifier in an hour, and its outputs become the training set for the small model later. That path, a big model labelling data for a small one, is in its simplest form.
What the first version got wrong was not the model choice; it was not noticing that the task had a shape, and that the shape matched a much cheaper tool.
Questions to ask
- Is the output a fixed set of values or free text?
- Do I already have labelled examples? (Queues that exist usually mean yes.)
- Do I need a confidence, and what happens below it?
- Could the expensive model label data now so a cheap model can take over later?
Mindset
Match the model to the shape of the output. Reading-and-deciding is a different job from writing, and the tool for it is smaller, faster, cheaper and more honest about its confidence.
Where it connects
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.