Pin Your Prompts to a Regression Suite Before Switching Models

A model upgrade is a one-line change that alters everything downstream of the prompt. The failures are quiet: a field that stops appearing in the JSON, a tool that stops being called, a refusal where there never was one, an answer that is fine but 30% more expensive. This is for teams running an LLM feature in production who want a switch to be a decision, not a gamble.

What actually changes underneath you

Start with what doesn’t move. A Claude model ID is a pinned snapshot: “When you use a model ID in an API request, the underlying model remains constant for the lifetime of that ID” (Anthropic (opens in a new tab)). From the 4.6 generation onwards the IDs are dateless (claude-sonnet-5), and a dateless ID is still a fixed snapshot rather than a pointer to the latest release. Older models carry a date (claude-haiku-4-5-20251001) and also accept a short alias that “resolves to the most recent dated snapshot for that minor version”: an alias can move under you, and a pinned ID can’t.

Two things move anyway. Serving infrastructure around the model changes: “Occasionally, infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed” (Anthropic (opens in a new tab)). And models retire. Anthropic gives “at least 60 days’ notice before model retirement”, after which requests fail; OpenAI promises at least six months for generally available models, three for specialised variants, and as little as two weeks for preview models (Anthropic (opens in a new tab), OpenAI (opens in a new tab)). You will move models whether or not you want to. The only question is whether you find out what changed before your users do.

Some of what changes is an outright API error, which is the easy half. Moving from Claude Sonnet 4.6 to Sonnet 5, sampling parameters (temperature, top_p, top_k) set to non-default values return a 400, manual extended thinking (thinking: {type: "enabled", budget_tokens: N}) returns a 400, and assistant-message prefill returns a 400 (Anthropic (opens in a new tab)). Your integration tests catch those the first time they run.

The expensive half is the changes that return HTTP 200. On the same migration, adaptive thinking is on by default, so a response “can begin with one or more thinking blocks before the first text block”, and anything reading content[0].text now reads an empty string instead of the answer. Refusals arrive as a successful response with stop_reason: "refusal". And the tokenizer changed: “The same input text produces approximately 30% more tokens than on Claude Sonnet 4.6”, which moves your token counts, your context headroom and any max_tokens you tuned by hand.

Build the suite from traffic you already have

Thirty cases from your logs beat three hundred invented ones. Take the requests your feature actually receives, keep the awkward ones (empty input, a 40-page document, the customer who writes in two languages), and store each as a case with the inputs your prompt takes.

Then decide what you assert. Exact-output matching is useless here: two models will phrase the same correct answer differently, and the suite will scream on every run until someone stops reading it. Assert the contract instead:

• the response parses, and required fields are present with the right types;
• the tool you expected was called, with arguments in range;
• forbidden content is absent (no invented IDs, no placeholder text, no refusal);
• the answer contains the facts the case requires, checked by substring, a regular expression or a model grader for the fuzzy ones;
• cost and latency stay within a band you set.

Some of those belong in the API rather than in assertions. Setting strict: true on a tool “guarantees Claude’s tool inputs match your JSON Schema by constraining the model’s token sampling to schema-valid outputs”; without it, “Claude might return incompatible types ("2" instead of 2) or omit required fields” (Anthropic (opens in a new tab)). Guarantee what you can, and test what’s left.

import Anthropic from "@anthropic-ai/sdk";
import { summarizeTicket } from "@/lib/ai/summarize-ticket"; // your production call, model injected

export interface Case {
  id: string;
  input: { ticketId: string; body: string };
  /** Facts the answer must contain, as they'd appear in the text. */
  mustMention: string[];
}

export interface CaseResult {
  id: string;
  parsed: boolean;
  missingFacts: string[];
  toolCalled: string | null;
  refused: boolean;
  outputTokens: number;
  ms: number;
  text: string;
}

export async function runCase(model: string, testCase: Case): Promise<CaseResult> {
  const started = Date.now();
  const message = await summarizeTicket(testCase.input, { model });
  const text = message.content
    // Not content[0]: with thinking on, thinking blocks come first.
    .filter((block): block is Anthropic.TextBlock => block.type === "text")
    .map((block) => block.text)
    .join("");
  const toolUse = message.content.find((block) => block.type === "tool_use");

  let parsed = true;
  try {
    JSON.parse(text);
  } catch {
    parsed = false;
  }

  return {
    id: testCase.id,
    parsed,
    missingFacts: testCase.mustMention.filter((fact) => !text.toLowerCase().includes(fact.toLowerCase())),
    toolCalled: toolUse && "name" in toolUse ? toolUse.name : null,
    refused: message.stop_reason === "refusal",
    outputTokens: message.usage.output_tokens,
    ms: Date.now() - started,
    text,
  };
}

The example assumes your production call takes the model as an argument, which is the one refactor worth doing before any of this: a single place where the model ID is chosen, and never an alias.

Read the diff, not the score

Run every case against the model you’re on and the model you’re considering, then compare per case. A single pass rate hides the two things you need: which cases changed, and in which direction.

Sort the results into three piles. Contract failures (unparseable JSON, a missing required field, a tool that wasn’t called, an unexpected refusal) block the switch until fixed. Content differences need a human or a model grader to judge: often the new answer is better, sometimes it drops a caveat your legal team asked for. Cost and latency shifts are a business decision: with a tokenizer change of roughly 30%, the same work can cost more even when the per-token price is lower, so re-baseline cost per customer rather than comparing price lists.

One more reason to look per case: models are not deterministic, so a single run tells you little about a case that flips. Run each case a handful of times and compare how often it passes, not whether it passed once; Measure Agent Reliability With pass^k covers the arithmetic and the traps. If the two models disagree on a case that flips under both, you’re looking at noise, not a regression.

Keep the transcripts. When a case regresses three weeks later, the stored request and response tell you whether the prompt drifted, the retrieved context changed, or the model did.

Before you flip the switch

Work through the provider’s own migration notes with your suite as the checklist. For the Sonnet 4.6 to Sonnet 5 move, that means: select response blocks by type rather than position, remove budget_tokens thinking and sampling parameters, drop assistant prefills, handle stop_reason: "refusal", re-run token counting instead of reusing old counts, and revisit max_tokens values tuned for the old tokenizer (Anthropic (opens in a new tab)). Then roll out behind a flag, with the old model one config change away.

Two habits make the next switch cheaper. Pin the model ID in one place, so “what are we running?” is a lookup and not an investigation; Anthropic’s Console can export usage by API key and model, which is how you find the job still pointing at something deprecated (Anthropic (opens in a new tab)). And run the suite on a schedule against the model you’re already on: the ID is pinned, but the infrastructure around it isn’t, and a weekly run turns “something feels off” into a dated diff.

The rule worth adopting: no model change ships until the suite has run on both models, every contract failure is fixed or accepted in writing, and the cost per request has been measured on your own traffic rather than read off a pricing page.

Sources

• Model IDs and versioning (Anthropic) (opens in a new tab)
• Model deprecations (Anthropic) (opens in a new tab)
• Migrating to Claude Sonnet 5 (Anthropic) (opens in a new tab)
• Strict tool use (Anthropic) (opens in a new tab)
• Deprecations (OpenAI) (opens in a new tab)