Phase 4 of 4
Systems
How do I run this in production?
Loops, tools, queues, budgets and the failure modes that only show up under load. Ordinary distributed systems work, with one unusual component in the middle.
Topics in this phase
Agents & tool use
7 minUnreadA loop where the model picks a tool and your code decides whether to run it, and every interesting problem is in the loop rather than in the model.
After Structured output, Evals
Workflow patterns
6 minUnreadFour shapes for composing model calls, ordered by how much control flow you hand over — and the right default is the one that hands over the least.
After Agents & tool use
Serving & batching
6 minUnreadDecoding is bandwidth-bound, so one read of the weights can serve many requests at once — and that single fact is what makes the economics work.
After Prefill & decode, KV cache
Cost & latency levers
6 minUnreadSix levers, three of which need no infrastructure at all and are almost always tried last.
After Prefill & decode, Context engineering
Observability
6 minUnreadAggregate metrics tell you something regressed; only a trace of the whole chain tells you which of six stages did it.
After Agents & tool use, Evals
Failure modes
6 minUnreadThe failures that matter here do not raise exceptions — they return 200 with the wrong thing — so the defences are structural rather than reactive.
After Observability, Agents & tool use
Examples that open after this phase
Problem statements that make sense once these topics are done. They unlock at the same 80% mark the next phase does.
An agent that books travel
Sort the tools by what happens if the model is wrong, and every control in the system lands on the one row where money moves.
after phase 4 · 4 min · #agents #security #risk
The 3am bill
A loop with no bound did not error; it worked all night, expensively, on a task it had stopped making progress on at step two.
after phase 4 · 4 min · #agents #cost #observability
The nightly job is making the chat slow
Throughput and latency pull in opposite directions under load; one serving pool cannot be tuned for both, and the fix is a capacity plan rather than a setting.
after phase 4 · 4 min · #serving #latency #capacity
It got worse and nothing changed
Every dashboard you already had was green; the one that moved is one you would only have if you built it for this kind of system.
after phase 4 · 4 min · #observability #debugging #retrieval
The poisoned PDF in the knowledge base
Text in a document reaches the model as text, indistinguishable from an instruction you wrote — so the defence is not a better prompt but a check at the point an action is taken.
after phase 4 · 4 min · #security #agents #retrieval
Cutting the bill by 80% without touching the model
The proposal was to reserve GPUs; the actual fix was three prompt changes and a router, applied in order of cost, and the GPUs were never needed.
after phase 4 · 4 min · #cost #prompts #serving