Asking for JSON in a prompt produces three problems: prose around the object, mis-escaped quotes, and invented enum values. Parsing crashes at 3am and whoever carries the pager finds out after. Structured outputs solve this at the root: you constrain generation token by token against a JSON Schema.

Constrained decoding

Constrained decoding restricts output against your schema at the token level. OpenAI structured outputs and Anthropic tool use guarantee syntactic validity from the first token to the last. On the open-source side, Outlines and llama.cpp grammars compile the schema into token masks and do the same job on your hardware. OpenAI exposes it through response_format with json_schema and strict: true; Anthropic forces a shape by combining tool definitions with tool_choice.

Schema design

  • Flat structures beat deep nesting; each extra level raises the odds of confusion
  • Enums beat free strings; models fill ambiguous fields with plausible wrong values
  • Explicit required fields plus additionalProperties: false; models respect constraints better than prose instructions

Semantic validation

Constrained decoding covers shape. A checkout date earlier than the order creation date sails through the schema and detonates in the accounting report. Coherent date ranges, referential integrity, and business rules sit beyond what decoding enforces. A validation layer built on zod, pydantic, or JSON Schema validators catches whatever slips through, and its errors feed the retry loop.

Retries with feedback

Validation failed? Send the prompt again with the validator message included, plus the rejected output so the model sees what to fix. One retry clears most cases. Three consecutive failures point at a sick prompt or schema and deserve a fallback path or a human review queue. Without that ceiling, the retry loop burns tokens over a deterministic bug.

Versioning

Field additions keep backward compatibility: responses cached weeks ago still validate against the new parser. Add fields at the end of the object, mark them optional, promote them later. A version number on the payload envelope helps stage migration cohorts.

Edge case tests

Property-based tests generate adversarial prompts probing boundaries: empty enums, boundary dates, strings packed with escape characters. Fold them into eval suites next to quality metrics. Format regression counts as regression.