Structured Outputs: From JSON Mode to B-Trees

“Just use JSON mode,” they say.

Anyone who has run JSON mode in production knows the truth: it leaks. It hallucinates keys. It truncates arrays.

Theoretical Limits of Instruction Tuning

Asking a model to “please output JSON” is just instruction following. It’s probabilistic. At temperature 0.7, there is a non-zero probability that the model decides to print a Markdown code block around your JSON, breaking your parser.

The Solution: Constrained Sampling

The real engineering solution is Constrained Sampling (available in llama.cpp via GBNF grammars, or via outlines in Python).

Instead of asking the model to be nice, we fundamentally alter the sampling step of the inference engine.

1. Define Grammar: You provide a CFG (Context-Free Grammar).
2. Mask Logits: At each step of generation, the engine masks out any token that would violate the grammar.

If the grammar expects an integer, the Model literally cannot output a letter. The probability of generating a letter becomes 0.

Implementation

Tools like Outlines and Guidance implement this today.

import outlines

model = outlines.models.transformers("mistralai/Mistral-7B-v0.1")
generator = outlines.generate.json(model, UserSchema)

# Guaranteed to be valid JSON matching UserSchema
result = generator("Create a user profile")

Stop relying on prompt engineering. Start using engineering.