Regression testing prompts and skills

Test the outcome, then locate the failure

Changing a prompt can fix one answer while breaking a later step. Repeating every complete workflow after every edit can be slow and expensive. Use cheap checks of contracts and selection first, then focused model tests, connected steps and a small retained set of complete workflow checks.

This strategy accompanies the component blueprint and its support-priority example. It is a proposed testing approach; the example counts and expected outcomes below are design choices, not measured results. The example owns the policy and expected handovers. This guide owns what to rerun after a change, how much to spend and what to record before release.

A regression test checks that a change preserves required behaviour. An evaluation case defines inputs, an environment and acceptance criteria. A trial is one attempt at that case. A grader checks the result using code, a model or human judgment. Version the cases, acceptance criteria and graders alongside the system. Retain each trial as an immutable record linked to the exact configuration and environment it exercised.

Give each case a known initial state, required rules and sources, permitted effects, expected outcomes, grader and risk category. Keep expected answers and grading instructions outside the model’s task context. Separate cases used to tune prompts from held-back cases used to assess a candidate. Assign related documents or incidents to the same set so near-duplicates cannot appear in both.

Use several layers of tests

Record whether a failure occurred in source extraction, retrieval, rule selection, context assembly, model application, validation or action execution. Supply known-correct context as a diagnostic: success with it points towards selection or assembly; continued failure calls for examining the model step and its contract. This diagnostic does not replace the complete workflow test.

Check source omission as well as answer support. An answer can be faithful to every retrieved passage while missing the decisive source. This is why retrieval metrics and answer grading need separate expected source sets. RAGAs and ARES are evaluation frameworks that provide research approaches to assessing retrieval-based systems. ARES also uses human annotations to calibrate assessment.

Work through a prompt change

In the support example, the policy, skill and output schema stay fixed. The priority prompt changes from version 3 to version 4. The intended improvement is more reliable attention to exceptions; the business rule does not change.

Old instruction excerpt:

Choose the best matching priority rule. Return the recommendation and sources.

Candidate instruction excerpt:

Evaluate the applicable rules and their declared exceptions before proposing
a priority. Report the applied rule and supporting source locations. If a
required fact is missing or conflicting, return the policy's assessment route.

These excerpts illustrate the edit; they are not complete deployable prompts. The comparison must retain the rest of the prompt, supplied context, model settings and environment. Whether the edit improves performance needs trials.

For the central case, the grader expects priority 1 under SP-R3 (Emergency access exception). It also expects the maintenance and emergency-route source fields and an unchanged ticket. A valid response format alone is insufficient.

case: maintenance_with_route_failure
input_fixture: ticket_T17_revision_4
required_rules: [SP-R1_v1, SP-R2_v1, SP-R3_v1]
required_sources: [T17_revision_4, M8_revision_2, I9_revision_1]
expected:
  priority: 1
  applied_rule: SP-R3
  supporting_locations: [T17.service_status, M8.status, M8.window, I9.route_status]
  final_ticket_revision: 4
forbidden_effects: [ticket_write, external_message]

Build an initial case set around meaningful differences:

Add variations that move the decisive passage to the middle, insert irrelevant material, paraphrase the request and change source dates. If the workflow summarises history or resumes a saved task, check that rule references, approval scope and unresolved obligations survive. Expected outcomes should stay stable where meaning is unchanged. Also change a decisive fact, such as route availability, to verify that the answer changes when it should.

These cases do not all require a model call. Missing rule dependencies should fail during configuration checks. In the worked design, missing or conflicting required facts take the controller’s assessment route before the model call. Test those routes with fixed inputs. A model test with deliberately incomplete context is an additional diagnostic, not proof that the real selector works. The timeout and stale-revision cases apply to the optional write operation.

The requirement is correct behaviour, not a fixed sequence of model thoughts or tool calls. Check intermediate steps when they implement a real obligation, such as reading current policy or obtaining approval.

Select regression tests from the change

Maintain a dependency map from prompts, skills, rule sets, source families and tool contracts to the cases that exercise them. Combine declared dependencies with observed traces. Old traces alone omit branches the new prompt may take.

For the version 3 → 4 change:

1. Check the configuration, references and output schema locally. Confirm that rule selection still includes the exception dependency.
2. Run both prompt versions against the same frozen inputs for cases that reach the decision model. Compare each outcome, including previously passing cases that now fail. Run retrieval independently where its selection is in question.
3. Pass actual candidate outputs into the validator and analyst response. Check both accepted and rejected proposals. Exercise the controller’s assessment route and prove that draft-only tasks cannot write.
4. Run a small complete-workflow set covering ordinary maintenance, the exception, missing information and hostile source text. If approved writes are enabled, include the gateway, stale state and timeout recovery too. These are illustrative coverage choices, not a sufficient release threshold for every system.
5. Repeat variable or consequential cases under a declared trial budget. Add newly discovered failures to the regression set and check held-back cases before accepting the change.

Run a broader suite before release or when shared dependencies change. During operation, sample real failures and changing task distributions into reviewed, sanitised cases. Periodic broader checks catch gaps in the dependency map.

Save cost without hiding untested behaviour

Reuse unchanged extraction, indexing and deterministic checks when their full inputs, versions and configuration match. Frozen upstream outputs help isolate a changed step. Label these component tests: they do not demonstrate that live routing and retrieval still supply those inputs.

Tool replay can replace an expensive external call with a recorded response only when the request, arguments, relevant state and contract match. If the candidate makes a new request, use a simulator that supports it or a controlled service test. Do not force the old response into a different branch. Reset state between trials and prohibit effects on real users or records.

Cached model responses are historical results. They cannot count as fresh trials or establish run-to-run consistency. Provider caching of input processing may reduce cost while still producing a new response; record which kind of caching occurred.

Estimate calls for each version as cases × trials × expected model calls per case, then sum across the versions being compared. Estimate their cost from input and output token use and the relevant rates, then add grading, retrieval, service use and human review. For workflows of different lengths, calculate these estimates per case and sum them. Use measured call counts and usage when available. Limit calls, elapsed time and spend; stop a candidate early on a decisive failure. If the budget expires before necessary coverage, report an incomplete result rather than a pass.

For a small planning example, compare four decision fixtures (ordinary outage, maintenance with the route available, maintenance with it unavailable, and an expired approval) across two prompt versions with three trials each: 24 model calls. Add three complete advisory workflows (ordinary maintenance, the exception and hostile source text), two versions and two trials: 12 further calls if each follows the example’s single-call path. That is 36 planned calls, plus any model grading or retries within the configured limit. Missing dependencies and the controller’s missing-fact route add deterministic checks, not model trials. Retrieval, tool access and review still contribute time and cost. These counts illustrate budgeting; they neither establish adequate coverage nor report a measured saving.

Decide what the results establish

Use code for exact decisions, source identifiers, schemas, permissions and final state. Use a calibrated rubric, a set of criteria for judging meaning, completeness and explanation. Model graders need comparison with human judgments, disagreement review and protection from instructions embedded in the response they are judging. Human review remains useful where correctness is disputed or consequences are large. Anthropic’s agent evaluation guidance describes this mixture of graders and outcome checks.

Set acceptance rules before viewing the candidate scores. Block release on any observed critical policy or authority violation. For routine quality, agree an allowable regression margin, required coverage and cost/latency limits. Report trial counts, per-case changes and uncertainty; zero observed critical failures in a small set does not establish zero risk. Inconclusive comparisons need more trials or an explicit decision by the release owner.

Report first-attempt success separately from success after retries. If a task must work consistently, also report how often all repeated trials pass. τ-bench uses final-state checks and repeated trials to examine tool-using agents; its benchmark results do not establish your system’s reliability. Do not multiply component pass rates to estimate whole-workflow success: failures can share causes and affect later steps.

Keep representative cases separate from a set deliberately enriched with rare failures. Both matter, but their average scores answer different questions. Report escaped failures and failures caught by validators separately, alongside cost per accepted task, elapsed time and human review effort.

Record the change decision

Keep one comparison record for the candidate, linked to individual trials:

A deliberately invalid proposal injected to test the validator passes that test when it is rejected as required. A wrong proposal produced by the candidate is a model failure even if the validator catches it. Keep those observations distinct. If the change is accepted, retain the exact configuration and the previous version for rollback, then monitor the same outcome criteria during operation.

Tools to require

These capabilities may live in ordinary version control and test tooling. Use the worked acceptance table to assess a tool against a concrete chain. The first useful deliverable is a repeatable explanation of which outcomes a change preserved, broke or left untested, with its actual cost.