The nightly job is making the chat slow
In one line
Throughput and latency pull in opposite directions under load; one serving pool cannot be tuned for both, and the fix is a capacity plan rather than a setting.
The situation
One self-hosted model serves two things: an interactive support chat, and a nightly job that classifies the day's forty thousand tickets. The nightly job was moved earlier to finish before the morning shift. Now, between 05:00 and 07:00, chat users see time to first token climb from half a second to four.
Someone proposes lowering the batch size. Someone else proposes raising it. Both are right, for different workloads, which is the point.
The trade-off is structural#
lets many requests share one pass through the model, and because decoding is memory-bound, a larger means far more for almost the same per-token time. That is exactly what the nightly job wants: forty thousand tickets, no user waiting, maximise tokens per second, minimise cost.
But a new request arriving at a full batch waits for a slot. Under a heavy batch load, for the chat is set by the queue, not by the model, and the queue is full of tickets. The setting that makes the batch job cheap is the setting that makes the chat slow, and no value of it serves both.
makes it worse: a few long ticket classifications hold slots for hundreds of steps, and the chat user behind them sees the p99.
Two pools, two settings#
Run two deployments of the same model. The interactive pool: small batches, admission control that rejects rather than queues past a threshold, output tokens capped, prefix caching for the shared system prompt. Tuned for the first token.
The batch pool: large batches, deep queue, no latency target beyond "done by morning". Tuned for tokens per second. If GPUs are scarce, it can share hardware with the interactive pool by time — but never by slot.
The cost of the split is a second deployment to operate. The benefit is that each pool has a capacity plan that means something: the interactive one is sized for peak concurrent users, the batch one for volume per hour. Before the split, neither number could be stated, because both workloads were competing for one answer.
Questions to ask
- Which of my workloads has a user waiting, and which does not?
- What is the latency target for the interactive path, and is anything enforcing it?
- Is the batch job's cost per ticket what it could be at a large batch size?
- Can I state the capacity of each pool as a number?
Mindset
Throughput and latency are opposite ends of one dial. When two workloads want different ends, the answer is not a compromise setting; it is two pools, each with a plan you can write down.
Where it connects
Phase 2 · 7 min
Prefill & decode
One request is two different workloads with two different bottlenecks, and every latency number you care about belongs to one or the other.
Phase 4 · 6 min
Serving & batching
Decoding is bandwidth-bound, so one read of the weights can serve many requests at once — and that single fact is what makes the economics work.
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.