Blocking what it should, without blocking what it should not
OR-Bench sample, vendor defaults. Horizontal: benign prompts refused, the over-refusal this post is about. Lower is better.
Vertical: toxic prompts refused or rebutted, the expected blocking most products want to keep. Higher is better. Top left is the target.
0%
70%
5%
78%
10%
85%
15%
92%
20%
100%
Gemini 3.8 Flash
3.9% over-refusal, 77.0% blocked
Claude Opus 5.5
4.0% over-refusal, 84.8% blocked
GLM 5.3
9.1% over-refusal, 89.9% blocked
DeepSeek V4.1 Flash
12.5% over-refusal, 94.9% blocked
GPT-6 Astra
17.1% over-refusal, 90.7% blocked
benign prompts refused (over-refusal)
toxic prompts blocked (expected)
Measured 2026-09-24. 77 audited-benign and about 100 toxic prompts per model; each rate carries a margin of 5 to 9 points.