Analysis: getting machine-readable output from a language model, and why 'return JSON' is not a specification

The moment a language model stops writing for a person and starts writing for a program, a different set of requirements arrives. Prose can be approximately right. A field named total_amount cannot.

Most teams discover this in the same order. They ask for JSON in the prompt and it works. They ship. A week later the parser throws because the model wrapped the object in a code fence, or added a friendly sentence before it, or emitted a trailing comma. They add a cleanup function. It works. A month later a field that should have been null contains a plausible invented value, and nothing crashes at all, which is much worse.

Those two failures are different in kind, and conflating them is why the problem takes so long to fix.

Two problems wearing one name

Syntactic failure is output that does not parse or does not match the schema. It is loud, immediate, and fully solvable.

Semantic failure is output that parses perfectly and says the wrong thing. A confident date that was never in the document. A category that fits the enum but not the content. A total that does not match its line items.

Constrained decoding — the schema-enforced modes now offered by every major platform — eliminates the first category almost entirely. It works by restricting which tokens are permitted at each generation step so that output outside the grammar cannot be produced. This is a genuine step change from prompt-based requests, which are suggestions the model usually honours, and the difference shows in the tail: the one request in a thousand that used to fail is the one that pages someone.

It does nothing whatsoever for the second category, and there is a real risk in how that lands psychologically. Teams that adopt schema enforcement often stop validating, because the platform "guarantees" the schema. The guarantee is about shape. Every remaining question — is this the right value, do these fields agree with each other, is this date within the plausible range — is still yours.

The schema is part of the prompt

This is the most useful reframing available, and it is routinely missed. The model reads the schema. Field names, enum values, descriptions and ordering are all instructions, and they compete with the system prompt for influence.

Which means schema design choices have quality consequences.

Name fields the way you would describe them to a person. is_urgent produces better results than flag2. A description on the field — "true only if the customer states a deadline within 48 hours" — is often more effective than the same sentence in the system prompt, because it sits adjacent to the decision.

Use enums instead of free strings wherever the value space is closed. A free-text category field produces a long tail of near-synonyms that your downstream code has to normalise forever. An enum makes the decision at generation time.

Keep it flat. Deeply nested objects increase both the chance of structural error and the blast radius when one branch goes wrong. If the domain is genuinely hierarchical, two calls are often more reliable than one deeply nested response — and easier to evaluate, because you can see which stage failed.

Order fields so that reasoning precedes conclusions. A schema where a reasoning or evidence_quote field comes before classification produces measurably better classifications than the reverse, because the model generates left to right and the earlier field conditions the later one. This is the one place where a small amount of generated prose earns its cost inside a structured response.

Give the model a way out

The single most common cause of invented values is a schema that offers no legitimate alternative to inventing one.

If contract_end_date is a required non-nullable string and the contract does not state an end date, the model must produce something. It will produce something plausible. Nothing in the pipeline will object.

The fix is to make absence expressible: nullable fields, an explicit not_stated enum member, or a separate boolean that gates the value. Paired with a clear instruction that absence is an acceptable and expected answer, this removes a large share of hallucinated field values — and it costs one line of schema.

The same logic extends to confidence. A confidence enum with three values — high, medium, low — is more useful than a numeric score, because models are poorly calibrated on numeric self-assessment but reasonably consistent about coarse buckets. Route the low bucket to a human and you have converted an accuracy problem into a throughput problem, which is a much better problem to have.

Do not ask the model to do arithmetic

A schema that contains subtotal, tax, and total is asking for three values where one is derivable from the others. The model will fill all three, and they will occasionally disagree.

Extract the inputs. Compute the derived values in code. The same rule applies to date arithmetic, percentages, counts, and anything with a unit conversion. This is not a statement about model capability — it is that a deterministic calculation has an error rate of zero and a generated one does not, and there is no reason to accept the second when the first is available.

Validation and the single repair turn

Validate every response, including from schema-enforcing endpoints. When validation fails, the most effective response is one repair attempt that includes the specific error, not a blind retry. Sending back "the field priority must be one of low, medium, high; you returned 'urgent'" resolves the great majority of residual failures in a single additional turn.

Bound it at one. Unbounded retry loops on an adversarial or genuinely ambiguous input convert a visible failure into an invisible cost, and they suppress exactly the signal you need — that something about this input does not fit the schema you designed.

Beyond the retry, have a deterministic fallback: a default record marked for review, a rules-based path, or a queue for a human. A pipeline whose only failure mode is "try again" has no failure mode at all, which is a design flaw rather than a feature.

Evaluating structured output

Structured output is unusually pleasant to evaluate, and teams should exploit that.

Because the output is machine-readable, most checks are exact: schema validity, enum membership, required-field presence, cross-field consistency, and field-level accuracy against a labelled set. Per-field accuracy is the metric to track, not whole-record accuracy — a record scored as a single unit hides which field is degrading, and it is usually one field, not the whole extraction.

Track the null rate too. A sudden drop in nulls after a prompt or model change often means the model has started inventing values where it used to abstain, and it is invisible to any check that only looks at valid records.

The unglamorous summary

There is no clever technique here. Turn on constrained decoding. Design the schema as if the model will read it, because it will. Let it say "I don't know". Keep arithmetic in code. Validate anyway. Repair once, then fall back.

Every one of these is cheap, and together they are the difference between a model that is a component of your system and a model that is an occasional source of surprises inside it.