Turning invoices into JSON, and the field that got invented
In one line
Schema enforcement made every output parse and quietly created a bug — a required field the invoice did not have, which the model filled in anyway.
The situation
Supplier invoices arrive as PDFs and need to become rows in the accounts system. The first version prompts for JSON and parses it; seven per cent of outputs fail to parse, mostly on a stray sentence before the brace. The second version switches on with a : parse failures go to zero overnight.
Three weeks later, accounts notice that a batch of invoices carry VAT numbers that do not exist. The invoices themselves have no VAT number. The schema said the field was required, so the model provided one.
What enforcement guarantees, and what it does not#
works by masking out, at every step, any token that would violate the schema. The output is valid by construction: no preamble, no trailing comment, no misspelled enum. That is a genuine and large win — it deleted a retry loop and a class of pager alerts.
It guarantees shape. It says nothing about values. And it introduces one new hazard: a required field is a demand. When the input has no VAT number, the model cannot omit the field, cannot leave it empty, and cannot say "not present". The only legal continuation is a string that looks like a VAT number, so that is what it produces. The constraint did not cause a hallucination; it removed the option not to.
Designing the schema for honesty#
Make absence expressible. "vat_number": {"type": ["string", "null"]} with a
description — "VAT registration number if printed on the invoice, else null"
— and the fabrications stop, because null is now the cheapest legal answer for
a missing value.
Put a place to think before the values. A leading free-text "notes" or
"reasoning" field lets the model observe "no VAT number visible" before it
commits to the structured fields. Generation is sequential, so field order is
prompt design.
Ask for evidence next to extractions that matter. A "total_source" field
holding the quoted text the total came from makes the value checkable, and
makes a wrong one visible to whoever reviews the batch.
Then measure the right thing. Parse rate went to 100% and told you nothing
about accuracy. An over a of real invoices
with hand-checked values reports per-field accuracy — and would have shown
vat_number at 9% "present when it should be null" on day one.
Questions to ask
- Which fields can genuinely be absent, and is absence expressible in the schema?
- Does the schema give the model somewhere to reason before it commits?
- Am I measuring parse rate or field accuracy? They are unrelated.
- For values that matter, can the output point at where it found them?
Mindset
A schema is a type signature at the edge of a non-deterministic function. Enforcing it fixes the shape and leaves the meaning to you — and every required field is an instruction to produce something, whether or not it exists.
Where it connects
Phase 3 · 6 min
Structured output
Getting a schema-valid object back instead of prose, so the boundary between the model and your code stops being a parsing problem.
Phase 3 · 6 min
Evals
Automated tests for a component whose output is not stable, where the assertion is a threshold on a score and the suite is the only thing making changes measurable.
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.