Test a Prompt Before It Ships: an Eval Caught a 16% Regression

A shorter prompt dropped one rule. On 110 new messages it left out the order number 18 times (16%) and the old one never, yet one 40-row run showed no misses.

October 20, 2026

Testing a prompt before it ships means running the new version and the current one on the same fixed inputs and checking every answer against rules you write down. We did that for v2 of a support prompt, which shortened the system text and dropped the rule “always state the order number”, a regression (a change that makes the output worse) that the test was meant to catch. On 110 messages that no earlier run had used, v2 left the order number out of 18 answers (16%) and v1 out of none, a difference far beyond chance (Fisher’s exact test, p = 3.6 × 10⁻⁶). One earlier run of 40 messages showed no misses at all, so a single passing run does not clear a version.

TL;DR

• v2 of support-answer omitted the order number in 18 of 110 answers in a test fixed before the run, and v1 in 0 of 110 (p = 3.6 × 10⁻⁶).
• One of three earlier 40-row runs passed v2 on all 40 rows, and one of the three final sets alone would not have caught it.
• v1 stated the order number in all 275 test calls the demo’s script made.
• Six test runs of 15 to 20 rows cost $0.14, $0.12 of it the judge.

This guide builds on Prompt versioning, which sets up support-answer and its versions. v1 and v2 both run on GPT-6 Luna; the only difference is the system text. From the same directory of github.com/synthorai-io/use-cases:

cd use-cases/prompt-library
python evals.py                        # v2 against v1 on the orders and general sets, three runs each
python evals.py --set fresh --runs 1   # the test fixed before the run
python evals.py --read RUN_ID          # read back a run started in the console

The runs on this page are from October 19, 2026, and the judge model is Claude Sonnet 5.5, which is not one of the versions under test.

What goes into a prompt test?

A test set is a table of inputs and a list of checks. Each row gives values for the prompt’s variables, and optionally a user message and an expected answer. A test run calls the candidate version and a baseline version for every row and grades each answer with every check. The check types are:

A set holds up to 50 rows, and each check applies to every row of its set. That decides how you split a test. “The answer contains an order number” only makes sense for messages that give one, so the demo keeps those messages in their own sets:

The judge check is one model call per answer that returns PASS or FAIL. It costs money and it can be wrong, so the demo uses it for what a pattern cannot express and reports it apart from the pattern checks.

How do you run a prompt test from code or CI?

CI (continuous integration, the checks that run on every change before it merges) can run a prompt test through the API, and your code runs the test itself: it opens a run, calls both versions by prompt_version for each row, grades the answers, and writes the results back to the run. The console then shows the run like any other. The demo’s loop, shortened:

EVAL = {"X-Synthorai-Metadata": '{"source":"eval"}'}   # keeps test calls out of per-version stats

run = admin.post("/prompts/support-answer/eval-runs",
                 json={"set_id": set_id, "candidate": 2, "baseline": 1}).json()["data"]
results = []
for i, row in enumerate(rows, 1):
    entry = {"row": i}
    for side, version in (("candidate", 2), ("baseline", 1)):
        answer = call_by_version("support-answer", row, version, headers=EVAL)   # a Chat Completions call
        entry[side] = {"version": version, "answer": answer.text,
                       "checks": {"order number": bool(ORDER.search(answer.text)),
                                  "answers": judge(ANSWERS, row, answer.text)}}
    results.append(entry)
    admin.put(f"/prompts/support-answer/eval-runs/{run['id']}",
              json={"status": "finished" if i == len(rows) else "running", "done": i, "checked": i,
                    "passed": sum(all(e["candidate"]["checks"].values()) for e in results),
                    "results": {"rows": results}})

call_by_version is a call by id with prompt_version set, as in Prompt versioning, and judge is a call to the judge model that asks for PASS or FAIL. The source: eval label matters during a rollout: per-version totals on the Prompts page leave labelled calls out, so a test does not skew a comparison of live traffic.

The demo grades its pattern checks in Python. To grade them exactly as the console does, send the checks and answers to POST /api/v1/prompt-checks/grade, which applies contains, not_contains, regex, not_regex, equals_expected and json on the server with RE2 and returns a pass or fail per check and answer. A quick test of it returned what you would expect for (?i)refund against “REFUND by Friday” (pass) and not_regex (?i)refund.{0,40}\bby\b against the same answer (fail).

The console’s Run button works the same way from the browser: it calls the two versions per row and stores the results. One console run of the 40-row orders set (run 10 below) graded the candidate only and showed v1’s answers next to it without checks.

Did the test catch the regression?

The test caught it. We fixed the test before running it: 110 new messages that no earlier run had used, one call per version per row, and a single Fisher’s exact test on the order-number check over all 110 rows, at a 0.05 threshold. Fisher’s exact test gives the probability of a difference at least this large if both versions really behaved the same; two-sided means it counts a difference in either direction. We committed the rows and this rule to the repository before the first call. The result:

v2 misses the order number in 16% of answers. The misses are spread across topics, not one kind of message: 4 of 22 each about delivery, returns, wrong items and damaged items, and 2 of 22 about billing. A typical miss is the answer to “Order A-50918 shows two pending charges in my bank app.”: it explained that the two entries may be pending authorizations rather than two charges, which the judge passed, and never named the order.

The per-set p values describe the three sets; they are not the test. Two of the three sets would have flagged v2 on their own and the third would not have.

Why can one passing run hide a regression?

A rule that a version breaks only some of the time can go unbroken for a whole run. Before the test above, the demo ran v2 against v1 three times on a 40-row set:

v2 passed all 40 rows in run 9, so a team that ran its test once and got that run would have shipped v2. None of these runs is significant on its own either: against a baseline that never misses, a run of 40 rows or more needs at least 6 misses to get below 0.05 (34 of 40 gives p = 0.026; 35 of 40 gives 0.055). Smaller sets need fewer: 5 misses out of 20 already give p = 0.047.

That gives a rule for sizing a test: use enough rows that 6 misses are likely, not only expected. At a miss rate of 16%, 48 rows give an 80% chance of at least 6 misses; 38 rows give only 58%, although 6 is the expected count. At 6.25%, the rate on the 40-row set above (10 misses in 160 answers), an 80% chance takes 126 rows. For a rule you care about, use more rows than one set holds, split them over several sets, and decide in advance how you will combine them. Repeating the same rows does not add evidence the way new rows do: the runs test the same messages again, so they are not independent samples, and in the demo three rows that v2 missed in one run it missed in a second.

Did the shorter prompt break anything else?

The other checks found nothing else broken. On the general set, three runs each:

• No refund date: neither version ever wrote “refund by” or “business days”. The judged check failed at most 2 of 20 answers per version per run, with no difference between versions.
• Latency: v1 went over 20 s on 2 rows in one run; v2 never did.
• Answers the question (judged): v2 rated higher in every set, 88 of 110 against 75 of 110 on the fresh sets. We did not fix this comparison in advance, so it stays a direction: by the judge’s reading, the shorter prompt answers more directly.

The console’s judge and the demo’s judge use different prompts (the console’s writes a reason). On the same 40 messages, with answers from different runs, the console passed v2 on 40 of 40 and the demo’s judge on 32 to 35. Compare judge results only within one grader.

What does a prompt test cost?

About a tenth of a cent per row for two versions and one judged check, with the models in this demo:

The judge is most of the cost because each answer gets its own judge call on a larger model. A run of 40 rows took 10 minutes with calls made one after another; parallel calls would cut that, at the cost of more load on your rate limit.

How does a test inform shipping?

When you move production in the console, the dialog shows the version’s last test run next to the change. It informs the move and does not block it. Test sets and the answers of a run are stored encrypted, and answers are kept 30 days.

The dialog counts a row as passed only when every check of the set passed, so the 28 of 35 there combines the pattern and the judge.

What is left out?

• Checks per row. A check applies to every row of its set, which is why the demo has separate sets for messages with and without an order number.
• Server-run tests. Both the API and the console’s Run button make the model calls from the caller’s side; there is no endpoint that runs a whole test on the server and waits for it.
• Judge accuracy. We did not check the judge against human ratings, so its numbers are directions, and the page claims a finding only for the pattern check.

FAQ

How do I test an LLM prompt before deploying it?

Run the new version and the current one on the same fixed inputs, check every answer against written rules, and compare pass counts with a significance test decided in advance. In the demo, 110 new messages and one Fisher’s exact test separated a 16% miss rate from 0%.

How many test cases do I need for a prompt regression test?

Use enough rows that at least 6 failures of the rule are likely if the regression is real. For an 80% chance, that is 48 rows at a 16% failure rate and 126 rows at 6.25%. In the demo one run of 40 rows passed a version that missed the rule in 16% of the answers in the main test.

Can an LLM judge replace pattern checks?

A judge should not replace a check that a pattern can express. In the demo the console and the script used different judge prompts and passed v2 on 40 of 40 and 32 to 35 of 40 of the same messages, while a pattern gives the same result for the same text every time.

Related reading

• Prompt versioning: calling a prompt by id and moving a label.
• A/B test a prompt change: what a version costs on live traffic.
• Prompt library guide: test sets, checks and eval runs.
• Prompts in the console: the test editor and the move dialog.

← Use cases