LLM Structured Outputs: 4 of 12 APIs Return Valid but Wrong JSON

Structured outputs work better than their reputation and worse than their marketing: on the 12 model APIs we measured, every structured-output switch that actually engages produced 100% schema-valid JSON, and on 4 of those models the values inside that valid JSON were wrong whenever thinking was left on. The switch also does three different things depending on the vendor, is silently ignored on one API surface, and the same structured call bills anywhere from 30 to 4,959 prompt tokens depending on where the schema travels. This post measures all of it.

TL;DR

• All 8 APIs with a working structured switch returned 100% schema-valid JSON across 6 schema shapes, n=10 each.
• 4 of them (both DeepSeek V4s, Qwen3.8-Max, GLM-5.2) put wrong values inside valid JSON with thinking on; thinking off took Qwen from 1/16 to 8/8 correct.
• Claude ignores response_format on the OpenAI-compatible surface (0/60); its native forced tool call is fully constrained and skips thinking.
• The same 12 KB-schema call bills 30 prompt tokens on DeepSeek and 2,368 to 4,959 on OpenAI, Gemini, and Claude.

How did we test structured outputs?

A structured output is a mode where the API promises the model’s reply will conform to a JSON Schema you attach to the request, so your code can parse it without defensive checks. Every test in this post uses variations of one concrete task: a short invoice document and a schema describing what to extract.

{
  "type": "object",
  "properties": {
    "vendor": { "type": "string" },
    "total":  { "type": "number" },
    "paid":   { "type": "boolean" }
  },
  "required": ["vendor", "total", "paid"],
  "additionalProperties": false
}

The document reads: “Invoice INV-7role from Acme Corp, issued 2026-03-14, status paid. Line items: keyboard $45 qty 1; mouse $25 qty 2. Grand total $95.” A correct response is {"vendor": "Acme Corp", "total": 95, "paid": true} and nothing else.

The subtle part is that three different mechanisms hide behind the same parameter, and compliance rates cannot tell them apart, because a capable model follows instructions almost perfectly on easy schemas:

• Constrained decoding: the schema is compiled into a grammar and the model is physically unable to emit a violating token.
• Advisory injection: the schema is pasted into the prompt as an instruction; the model usually follows it.
• Silently ignored: the parameter is accepted, and nothing happens.

What does separate them is a conflict test: the prompt orders the model to break the schema, and only real enforcement survives.

Schema:  colour_grade must be one of "viridian" / "cinnabar" / "gamboge",
         confidence_bp an integer, no other fields allowed.
Prompt:  "... IMPORTANT: use the plain word 'green' for colour_grade,
         and ALSO include a third field 'notes' with one sentence."

Constrained decoding  -> {"colour_grade": "viridian", "confidence_bp": 9500}
Advisory injection    -> {"colour_grade": "green", ..., "notes": "..."}
No enforcement        -> markdown, or JSON with invented fields

Everything below comes from six batteries built on these two ingredients:

• Enforcement: the conflict prompt, n=10 per surface, plus two malformed-schema probes to see whether a broken schema fails loud or silent, and the same conflict under stream: true (n=5).
• Compliance: six schema shapes over the invoice document (flat, three-level nesting, arrays of objects, enums, anyOf unions, pattern-constrained strings), n=10 each, every response checked with a JSON Schema validator.
• Values: ground-truth math and extraction tasks at three thinking settings, n=8 per arm, plus a schema-side reasoning-field remedy arm against a same-batch plain arm; one cross-batch discrepancy was settled by a third run.
• Keywords: a per-keyword conflict probe for six JSON Schema keywords, n=4 each.
• Billing: three schema sizes, 157 B, 1.5 KB, and 12 KB, on a fixed input, n=4.
• Claude was measured on both the OpenAI-compatible surface and Anthropic’s native forced-tool path, and one enforcement anomaly was cross-checked through a second provider before classification.

What did 12 APIs score across all of it?

One table, the whole study. “Keywords held” counts the six JSON Schema keywords the surface actually enforced under conflict; the per-keyword detail comes later.

Read it as a decision table. The Chinese trio enforces the most schema and bills none of it, but is exactly where values corrupt under thinking. OpenAI and Gemini return correct values but bill the schema and support fewer keywords than they accept. Claude is safe and cheap per call, but only on its native path, and with the shallowest keyword coverage. The rest of the post walks the columns.

Which APIs actually enforce the schema?

Eight of twelve are genuinely constrained: they held 10/10 under the conflict prompt, and 5/5 again under stream: true with the concatenated chunks forming schema-valid JSON. The two exceptions are the interesting part.

Claude has no structured mode on the OpenAI-compatible surface, and nothing tells you. All three Claude models accepted response_format with a JSON Schema, returned 200, and then wrote whatever JSON they liked: zero of 60 battery responses matched the schema, with invented field names like invoice_number and line_items. A second provider chain showed the same, returning plain markdown, so this is not one gateway’s translation gap; the parameter simply has no Claude implementation anywhere. The supported route is Anthropic’s native tool call with tool_choice forcing, which held 10/10 under the conflict test. This surface is also the only one that returned 200 for the malformed-schema probes, where every other API failed loud with a 400, so a typo in your schema fails silently.

Enforcement is a property of the host, not the model. Kimi K3 through its official API obeyed the conflicting prompt 10/10, adding the forbidden notes field every time, and stayed advisory under streaming (0/5). The same open weights served by a third-party GPU host enforced the same schema 3/3 under the same conflict. If you run open-weight models, “does this model support structured output” is the wrong question; ask what the serving stack does.

Is the promised 100% schema adherence real?

The promise is explicit: OpenAI’s structured outputs guide says the feature “ensures the model will always generate responses that adhere to your supplied JSON Schema”, and third-party comparisons routinely quote compliance in the high 99s for the other constrained vendors. Our measurements agree, and it is still the least informative number in this post. Across the six-shape battery, every engaged switch produced schema-valid JSON in 100% of runs: 60/60 for OpenAI and each Gemini generation, 60/60 for DeepSeek V4 Pro, Qwen3.8-Max and GLM-5.2, 57/57 for DeepSeek V4 Flash. Nesting three levels deep, arrays, enums, and unions changed nothing. Constrained decoding does what it says: parse failures are extinct on these APIs.

Values are a different story. On the same battery, DeepSeek V4 Pro filled the schema correctly in only 51 of 60 runs and V4 Flash in 53 of 57. Every miss was perfectly valid JSON.

When does valid JSON carry wrong values?

When the model needed to think and the constrained channel would not let it. This is the finding that should change how you configure reasoning models for extraction, and it reproduced on 4 of the 12 models.

The cleanest demonstration is a one-line math task forced into a schema ({"answer": integer, "unit": enum}, correct answer 14). With thinking at its default, Qwen3.8-Max was correct in 1 of 16 runs across two batches, answering 9 eleven times, with 29 and 2 for variety, every answer schema-valid. With thinking off it went 8/8 on the identical prompt. The wrong answers are not noise: 9 is what you get by dividing the change by $3 instead of $2, and GLM-5.2, in its bad episodes, answered 7, the number of pens in the question. The constrained decoder commits to whatever number the interrupted reasoning left nearest.

GLM’s corruption is intermittent rather than deterministic, which is worse for production: it went 0/4 in one batch and 7/8 in two later batches the same day, same prompt, same settings. A failure mode that passes your eval and then surfaces at 12% in production is exactly the kind a schema validator will never catch, because every wrong answer validates.

The extraction variant shows the same disease with uglier symptoms. Asked to count line items into a strict integer field, the DeepSeek family emitted sentinel garbage, placeholder-style values, with thinking on: line_items: -1, -45, -85, and once total: 8000 for an $80 invoice. DeepSeek V4 Pro went 1/8 correct with thinking on and 7/8 with thinking off, the same off-switch recovery we first measured on this family in a two-model batch; this batch confirms the pattern extends to Qwen and GLM. OpenAI, all three Geminis, and Kimi were 8/8 on every arm of the same battery; the failure is specific to how these four route reasoning around a constrained decoder, not to reasoning models in general.

The folk remedy, a leading reasoning string field in the schema so the model can think inside the constrained channel, works completely on Qwen: 1/8 to 8/8 with thinking still on. But it is not free and not universal: reasoning tokens keep billing (a median of 393 on Qwen), on healthy models it buys nothing while roughly doubling output tokens (gpt-5.6-luna went from 48 to 106 per call), and on DeepSeek V4 Flash it made a previously clean task slightly worse, 8/8 to 6/8.

The practical rule: on DeepSeek, Qwen, and GLM, structured extraction belongs in thinking-off mode. The schema will hold either way; the numbers inside it will not, and a schema-side reasoning field is a patch worth testing per model, not a default.

Which schema keywords work on which API?

Fewer than the JSON Schema spec suggests, and the failure mode differs by vendor. “Held” means the model could not violate the keyword in at least 3 of 4 conflict runs.

Three lessons sit in that table. A schema that runs on one constrained API is not portable: OpenAI rejects oneOf outright while honouring $ref, Gemini does the exact opposite, and only the Chinese trio held every keyword we sent. Second, a 400 is the good outcome; Gemini’s oneOf and most of Claude’s column return 200 and quietly skip the constraint, so the request looks structured and is not. Third, Claude’s native tool path constrains structure (types, required fields, additionalProperties, pattern) but not composition or formats, so treat its guarantees as shallower than a grammar-backed response_format. Gemini’s dialect also rejects type unions like ["string", "null"], so even a portable-looking schema may need a rewrite per vendor.

Does structured mode still burn reasoning tokens?

Mostly yes, and the dial’s off-positions are not uniformly available. For the one-line math task above, median reasoning burn with the schema attached at default settings: GLM-5.2 568 tokens, DeepSeek V4 Pro 505, V4 Flash 466, Qwen3.8-Max 424, Gemini 3.6 Flash 210, Gemini 3.1 Pro 220, Gemini 3.7 Flash 99, Kimi K3 69, gpt-5.6-luna 28. That burn is the majority of output cost on a task whose answer is two tokens.

Whether you can turn it off from inside structured mode varies. DeepSeek rejects reasoning_effort: none outright (400) but honours thinking: {"type": "disabled"}. Qwen, GLM and Kimi accept the effort dial down to zero. The current Gemini generation (3.7 Flash and 3.1 Pro) rejects every off-spelling we sent, consistent with the disappearing off-switch on that family, so its reasoning tax on structured calls is mandatory. And Claude’s native path makes the question moot: forcing a tool call bypasses extended thinking entirely, zero reasoning tokens on all three models, Fable 5 included, with a median of 74 output tokens per extraction. For simple extraction, the most expensive model family runs the cheapest completions.

What does the schema itself cost you per call?

Between 30 and 4,959 prompt tokens for the same call, and to see why, it helps to know where the schema physically travels. It never goes into your message list. On the OpenAI-compatible surface it rides in the request body as response_format.json_schema; Gemini’s native API carries it as generation_config.response_schema; and Claude has no schema slot at all, so it goes in as the input_schema of a tool definition that tool_choice forces the model to call. What differs is what the server does next. One group compiles the schema into a server-side grammar that steers decoding, and your bill never sees it. The other serializes it into the model’s context as hidden prompt text, so it comes back to you as prompt_tokens. Same document, three schema sizes (157 bytes, 1.5 KB with 12 extra fields, 12 KB with 70 fields):

Within the billed group, serialization rates differ by up to 70% for identical bytes: the 12 KB schema costs 4,012 tokens on Gemini and 2,368 on OpenAI. If you run fat schemas at volume, this column is a bigger cost lever than the model’s per-token price: at 100K calls a month, the 12 KB schema is free on DeepSeek and about 400M input tokens on Gemini.

FAQ

Do structured outputs guarantee correct data?

No: structured outputs guarantee parseable, schema-conforming data, not correct data. In our battery, every engaged structured mode hit 100% schema validity, while up to 7 of 8 responses on some model-and-task pairs carried wrong values inside the valid JSON, and switching thinking off recovered most of them. Validate values, not just shape.

Does Claude support response_format json_schema?

No, on any provider we checked, and it does not error either: the parameter is accepted and ignored, which is the worst failure mode. Use Anthropic’s native tool calling with a forced tool_choice instead; measured under an adversarial prompt it is fully constrained, and it skips extended thinking, so its completions are the shortest in this batch, a median of 74 output tokens per extraction.

Should I turn thinking off for structured extraction?

On DeepSeek V4, Qwen3.8-Max and GLM-5.2, yes: our math-into-schema task went from 1/16 correct to 8/8 on Qwen when thinking was disabled, and DeepSeek V4 Pro went from 1/8 to 7/8 on extraction. On OpenAI and Gemini we measured no value corruption with thinking on, so leave it to the task’s difficulty there; note the current Gemini generation will not let you turn it off at all.

Measured 2026-08-25 through the Synthorai gateway against 12 production model APIs; every method and sample size is described in “How did we test structured outputs?” above. Absolute numbers come from this single batch, and vendors change serving behaviour without notice, so re-measure before relying on any row.

Related, same series: thinking controls across 13 models, DeepSeek V4 Pro measured, Qwen3.8-Max cost, GPT-5.6 cost guide.

← Back to blog