Asking for JSON in a prompt returns three flavors of pain: prose wrapped around the object, quotes escaped wrong, invented enum values. Parsing crashes at 3am and whoever carries the pager finds out after. Structured outputs attack the root cause: instead of asking and hoping, you constrain generation token by token against a JSON Schema.

Constrained decoding: validity during generation

Constrained decoding constrains output against your schema at the token level. OpenAI structured outputs and Anthropic tool use guarantee syntactic validity from the first token to the last. On the open-source side, grammar-based engines such as Outlines and llama.cpp grammars compile the schema into token masks and do the same job on your own hardware. OpenAI exposes it through response_format with json_schema and strict: true; Anthropic forces a shape by combining tool definitions with tool_choice.

Schema design rules

  • Flat structures beat deep nesting; each extra level raises the odds of confusion
  • Enums beat free strings; models fill ambiguous fields with plausible wrong values
  • Explicit required fields plus additionalProperties: false; models respect constraints better than prose instructions

Validation stays your job

Constrained decoding covers shape; semantics remain your responsibility. A checkout date earlier than the order creation date sails through the schema and detonates in the accounting report. Coherent date ranges, referential integrity, and business rules sit beyond what decoding enforces. A validation layer built on zod, pydantic, or JSON Schema validators catches whatever slips through, and its errors feed the retry loop.

Retries with the error in hand

Validation failed? Send the prompt again with the validator message included, plus the rejected output so the model sees what to fix. One retry clears most cases; three consecutive failures point at a sick prompt or schema and deserve a fallback path or a human review queue. Without that ceiling, the retry loop burns tokens over a deterministic bug.

Schema versioning

Field additions need backward compatibility: responses cached weeks ago still have to validate against new parsers. Add fields at the end of the object, mark them optional, promote them later; a version number on the payload envelope helps stage migration cohorts.

Test the schema edges

Property-based tests generate adversarial prompts probing boundaries: empty enums, boundary dates, strings packed with escape characters. Aim prompts at enum boundaries and nested optionals; those two spots cause most production format breaks. Fold them into eval suites next to quality metrics; format regression counts as regression too.