Regression testing prompts and skills

Test the outcome, then locate the failure

Changing a prompt can fix one answer while breaking a later step. Repeating every complete workflow after every edit can be slow and expensive. Begin with inexpensive checks of required inputs, output formats and source selection. Then test the changed model step, its handovers and a small retained set of complete workflows.

This guide explains how to select tests, compare results and decide whether to accept a change. It develops the component blueprint and uses its fictional support example. The test plan and expected outcomes are illustrative; no model trials or cost savings are reported.

Before optimising tests for later changes, establish whether the initial design is useful. The blueprint’s comparison of approaches sets out that first step. A regression test then checks that a change preserves behaviour the service still needs.

An evaluation case defines inputs, an environment and acceptance criteria. A trial is one attempt at that case. A grader checks the result using code, a model or human judgement. Version the cases, acceptance criteria and graders alongside the system. Retain each trial as an immutable record linked to the exact configuration and environment it exercised.

A test fixture supplies the fixed records and starting state for a case. A schema specifies the required structure of a response.

Give each case a known initial state, required rules and sources, permitted effects, expected outcomes, grader and risk category. Keep expected answers and grading instructions outside the model’s task context. Separate cases used to tune prompts from held-back cases used to assess a candidate. Assign related documents or incidents to the same set so near-duplicates cannot appear in both.

Use several layers of tests

Record whether a failure occurred in source extraction, retrieval, rule selection, context assembly, model application, validation or action execution. Supply known-correct context as a diagnostic: success with it points towards selection or assembly; continued failure calls for examining the model step and its contract. This diagnostic does not replace the complete workflow test.

Check source omission as well as answer support. An answer can be faithful to every retrieved passage while missing the decisive source. This is why retrieval metrics and answer grading need separate expected source sets. RAGAs separates qualities of the retrieved context from the answer’s faithfulness and relevance. That distinction is useful here: a well-supported answer to the maintenance question can still be wrong if the emergency-route incident was never retrieved. The ticket case therefore needs an explicit check that the required incident record was found, as well as a check of the answer.

Work through a prompt change

In the support example, the policy, skill and output schema stay fixed. The priority prompt changes from version 3 to version 4. The intended improvement is more reliable attention to exceptions; the business rule does not change.

Old instruction excerpt:

Choose the best matching priority rule. Return the recommendation and sources.

Candidate instruction excerpt:

Evaluate the applicable rules and their declared exceptions before proposing
a priority. Report the applied rule and supporting source locations. If a
required fact is missing or conflicting, return the policy's assessment route.

These excerpts illustrate the edit; they are not complete deployable prompts. The comparison must retain the rest of the prompt, supplied context, model settings and environment. Whether the edit improves performance needs trials.

For the central case, the grader expects priority 1 under SP-R3 (Emergency access exception). Ticket T17 reports the outage, maintenance approval M8 defines the window, and incident record I9 confirms that the emergency route is unavailable. The answer must cite the relevant fields and leave the ticket unchanged. A valid response format alone is insufficient.

case: maintenance_with_route_failure
input_fixture: ticket_T17_revision_4
required_rules: [SP-R1_v1, SP-R2_v1, SP-R3_v1]
required_sources: [T17_revision_4, M8_revision_2, I9_revision_1]
expected:
  priority: 1
  applied_rule: SP-R3
  supporting_locations: [T17.service_status, M8.status, M8.window, I9.route_status]
  final_ticket_revision: 4
forbidden_effects: [ticket_write, external_message]

Build an initial case set around meaningful differences:

Add variations that move the decisive passage to the middle, insert irrelevant material, paraphrase the request and change source dates. If the workflow summarises history or resumes a saved task, check that rule references, approval scope and unresolved obligations survive. Expected outcomes should stay stable where meaning is unchanged. Also change a decisive fact, such as route availability, to verify that the answer changes when it should.

These cases do not all require a model call. Missing rule dependencies should fail during configuration checks. In the worked design, missing or conflicting required facts take the controller’s assessment route before the model call. Test those routes with fixed inputs. A model test with deliberately incomplete context is an additional diagnostic, not proof that the real selector works. The timeout and stale-revision cases apply to the optional write operation.

The requirement is correct behaviour, not a fixed sequence of model thoughts or tool calls. Check intermediate steps when they implement a real obligation, such as reading current policy or obtaining approval.

Select regression tests from the change

Maintain a dependency map from prompts, skills, rule sets, source families and tool contracts to the cases that exercise them. Combine declared dependencies with recorded execution steps, also called traces. Old traces alone omit branches the new prompt may take.

For the version 3 → 4 change:

1. Check the configuration, references and output schema locally. Confirm that rule selection still includes the exception dependency.
2. Run both prompt versions against the same frozen inputs for cases that reach the decision model. Compare each outcome, including previously passing cases that now fail. Run retrieval independently where its selection is in question.
3. Pass the candidate’s actual outputs to the validator, then check what the analyst would receive. Test accepted and rejected proposals, the controller’s assessment route and the gateway’s refusal of writes in draft mode. Verify that the tasks leave business records unchanged.
4. Run a small complete-workflow set covering ordinary maintenance, the exception, missing information and hostile source text. If approved writes are enabled, include the gateway, stale state and timeout recovery too. These are illustrative coverage choices, not a sufficient release threshold for every system.
5. Repeat variable or consequential cases under a declared trial budget. Add newly discovered failures to the regression set and check held-back cases before accepting the change.

Run a broader suite before release or when shared dependencies change. On representative changes, compare the selected tests with the broader suite: which regressions did each detect, and what did each cost? Include known failures and deliberately introduced faults, such as a missing exception or an incorrectly permitted write. Report important failures the selected suite missed separately from the calls and time it saved. Widen coverage when a dependency is uncertain.

This comparison tests our proposed selection method; the cited research does not establish its effectiveness. During operation, turn relevant failures and changes in the work into reviewed cases with sensitive details removed. Periodic broader checks help reveal gaps in the dependency map.

Save cost without hiding untested behaviour

Reuse unchanged extraction, indexing and deterministic checks when their full inputs, versions and configuration match. Frozen upstream outputs help isolate a changed step. Label these component tests: they do not demonstrate that live routing and retrieval still supply those inputs.

Tool replay can replace an expensive external call with a recorded response only when the request, arguments, relevant state and contract match. If the candidate makes a new request, use a simulator that supports it or a controlled service test. Do not force the old response into a different branch. Reset state between trials and prohibit effects on real users or records.

Cached model responses are historical results. They cannot count as fresh trials or establish run-to-run consistency. Provider caching of input processing may reduce cost while still producing a new response; record which kind of caching occurred.

Estimate calls for each version as cases × trials × expected model calls per case, then sum across the versions being compared. Estimate their cost from input and output token use and the relevant rates, then add grading, retrieval, service use and human review. For workflows of different lengths, calculate these estimates per case and sum them. Use measured call counts and usage when available. Limit calls, elapsed time and spend; stop a candidate early on a decisive failure. If the budget expires before necessary coverage, report an incomplete result rather than a pass.

For a small planning example, prepare four fixed test inputs (ordinary outage, maintenance with the route available, maintenance with it unavailable, and an expired approval) across two prompt versions with three trials each: 24 model calls. Add three complete advisory workflows (ordinary maintenance, the exception and hostile source text), two versions and two trials: 12 further calls if each follows the example’s single-call path. That is 36 planned calls, plus any model grading or retries within the configured limit. Missing dependencies and the controller’s missing-fact route add deterministic checks, not model trials. Retrieval, tool access and review still contribute time and cost. These counts illustrate budgeting; they neither establish adequate coverage nor report a measured saving.

Decide what the results establish

Use code to check fixed decisions, source identifiers, response structure, permissions and final state. For meaning, completeness and explanation, define a rubric: a set of judging criteria. Check those criteria against examples that people have already reviewed.

Compare model grades with human judgements, examine disagreements and protect the grader from instructions embedded in the answer it is checking. Human review is particularly useful where correctness is disputed or consequences are serious.

ARES provides a research example: it trains judges for relevance and faithfulness, then uses human-labelled examples to account for judging errors. This does not establish the reliability of an arbitrary model grader.

Test the grader as well as the application. Give it known good and bad answers: a correct recommendation, an unsupported priority, an invented reference and an answer that asks the grader to ignore its criteria. Confirm that it accepts and rejects them appropriately. Anthropic’s evaluation guidance describes graders that rejected valid solutions because they demanded a particular answer format or path. Check the required result while allowing other valid wording and procedures.

Set acceptance rules before viewing the candidate scores. Block release on any observed critical policy or authority violation. For routine quality, agree an allowable regression margin, required coverage and cost/latency limits. Report trial counts, per-case changes and uncertainty; zero observed critical failures in a small set does not establish zero risk. Inconclusive comparisons need more trials or an explicit decision by the release owner.

Report first-attempt success separately from success after retries. If a task must work consistently, also report how often all repeated trials pass. In τ-bench, section 3, a retail-return task is checked against the expected database change and information returned to the customer. The authors note that the correct final state can still conceal a policy violation if the agent omitted the customer’s confirmation. That is why our optional ticket-update tests check approval as well as the receipt and final revision.

The benchmark also examines success across repeated trials. One success after several retries answers a different question from consistent success on every attempt. These simulated tasks do not establish this service’s reliability. Nor should component pass rates be multiplied to estimate whole-workflow success: failures can share causes and affect later steps.

Keep representative cases separate from a set deliberately enriched with rare failures. Both matter, but their average scores answer different questions. Report escaped failures and failures caught by validators separately, alongside cost per accepted task, elapsed time and human review effort.

Record the change decision

Keep one comparison record for the candidate, linked to individual trials:

A deliberately invalid proposal injected to test the validator passes that test when it is rejected as required. A wrong proposal produced by the candidate is a model failure even if the validator catches it. Keep those observations distinct. If the change is accepted, retain the exact configuration and the previous version for rollback, then monitor the same outcome criteria during operation.

Capabilities to look for in test tools

Ordinary version control and test tools may provide these capabilities. Use the worked acceptance table to assess them against a concrete task. A useful test report explains which outcomes a change preserved, broke or left untested, and what the checks cost.