Structured output

In one line

Getting a schema-valid object back instead of prose, so the boundary between the model and your code stops being a parsing problem.

Why it exists

A model emits text. Your code needs a value. Between the two sits a boundary that, done naively, is a regex and a prayer: the model says "Sure! Here's the JSON you asked for:" and then mostly-valid JSON, and sometimes a trailing comment, and occasionally a markdown fence.

removes the boundary problem entirely by constraining generation so that only schema-valid continuations are possible. The difference between "usually parses" and "always parses" is the difference between a retry loop with a fallback path and an ordinary function call.

Illegal tokens are set to negative infinity before sampling, so the output is valid by construction. Valid is not the same as correct.

How constraining actually works#

At each step the model produces — one score per vocabulary entry — and samples from them. inserts one operation before the sampling: set the logit of every that would make the output violate the schema to negative infinity.

After softmax, those tokens have probability zero. They cannot be selected, not unlikely to be selected. The output is valid by construction rather than by luck.

Concretely, if the schema says status is an enum of "open" and "closed", then after emitting {"status": the only permitted next tokens are those beginning one of the two legal strings. After "op, the only legal continuation is en". The model is walking a grammar, and the sampling only chooses between branches the grammar allows.

This generalises past JSON — the same mechanism enforces any formal grammar, and it is how SQL-constrained and regex-constrained generation work. The cost is a small amount of per-step work to compute the allowed set, usually negligible next to the model pass itself.

is the same machinery wearing a different name. A tool definition is a for the arguments, and a tool call is a constrained generation against it. If you understand one you understand both.

Valid is not correct#

This is the distinction that catches teams, and it is worth being blunt about.

Constrained decoding guarantees your object parses and matches the schema. It guarantees nothing about whether the values are right. {"invoice_total": 4200, "currency": "EUR"} is perfectly valid and may be wrong in both fields.

Worse, constraint can actively produce wrong values. If a required field has no support in the input — the invoice genuinely has no VAT number — the model cannot decline, because declining is not a legal continuation. It will produce something schema-shaped. Required fields are a demand for a value, and the model will meet it.

Two design responses follow. Make fields nullable or optional where the answer may genuinely be absent, and include an explicit "not_found" enum member rather than forcing a guess. And separate extraction from judgement: ask for the fields plus a confidence or a supporting quote, so downstream code has something to gate on.

There is also a subtler effect worth knowing. Heavy constraint can reduce reasoning quality, because the model is denied the intermediate text it would otherwise use to work through the problem. The standard fix — a in miniature — is to put a free-text reasoning field first in the schema — field order matters, since generation is sequential — so the model thinks before it commits to the structured values.

Designing the schema#

The schema is an interface, and it deserves the attention you would give any other one.

Flat beats nested. Deeply nested objects are harder for the model to fill consistently and harder for you to validate incrementally. Two flat objects usually beat one three-level tree.

Enums beat free strings. Any field with a known domain should be an enum. It removes a normalisation step and it makes the constraint do real work.

Describe every field. Schema descriptions go into the prompt and are read. "amount": "total in minor units, e.g. 4200 for £42.00" prevents an entire class of unit error for the cost of a dozen tokens.

Name fields the way your domain does. The model has strong priors from training on ordinary code. customer_email behaves better than f3.

Order fields by dependency. Generation is left to right, so a field that should be informed by another must come after it. Reasoning first, conclusion last.

And validate anyway, on your side. Schema enforcement is a provider feature that can be unavailable on a fallback path, a different model, or an older API version. The is your own validator; the constraint is an optimisation that means it rarely fires.

Worked example

Extracting fields from support emails. Version one, prose instructions and examples, no enforcement:

parse failures                7.2%   of 10,000 emails
wrong enum spelling           3.1%   "Billing" vs "billing"
missing required field        1.8%
retry cost                    ~12% of total spend

Version two, the same prompt with an enforced schema:

{
  "reasoning":  { "type": "string" },
  "category":   { "enum": ["billing","technical","account","other"] },
  "urgency":    { "enum": ["low","normal","high"] },
  "order_id":   { "type": ["string","null"],
                  "description": "order reference if stated, else null" },
  "sentiment":  { "type": "number", "minimum": -1, "maximum": 1 }
}

parse failures                0.0%
wrong enum spelling           0.0%   structurally impossible now
missing required field        0.0%
retry cost                    0%

Extraction accuracy — whether order_id is the right order — went from 84% to 86%. The two points came from the field descriptions, not from the constraint. The constraint removed an entire class of engineering, and it did not make the model better at reading emails. Those are different wins and it is worth keeping them separate in your head.

One more detail from that run: order_id was initially a required string, and in 9% of emails there was no order reference at all. The model invented plausible ones. Making it nullable with an explicit description fixed a correctness bug that the schema had created.

Gotchas

Treating schema validity as answer correctness. The object parses; the values may be wrong. Evaluate field-level accuracy against labelled data, not parse rate, or you will ship a system with a perfect 0% error rate on the wrong metric.
Making everything required. A required field the input does not support is an instruction to fabricate. Nullable types and explicit "unknown" enum members are correctness features.
Forgetting to leave room for reasoning. Forcing a bare answer object on a task that needs working-out costs accuracy. A leading free-text field is cheap and usually recovers it.
Assuming enforcement everywhere. Not every model, endpoint or streaming mode supports it, and fallbacks quietly do not. Keep your own validator in the path and log when it fires.
Skipping field descriptions. They are the highest-leverage tokens in the whole schema, they live in the cached prefix so they are nearly free on repeat calls, and they are the difference between a schema that documents shape and one that documents meaning. Cover them in your fixtures too.

Mental model

A schema is a type signature at the boundary of a non-deterministic function, and constrained decoding is the compiler that makes the signature binding. It is exactly the guarantee a static type gives you: the shape is checked, the meaning is not. parseInt returning a number tells you nothing about whether it is the right number — and this is the same, at the edge of an rather than at the edge of a parser.

In practice

Problem statements that lean on this topic. Rated by the phase that makes them land.

Done reading?

Nothing marks itself complete. Say so only when you could explain this to someone else.

Next: Evals