Measure Agent Reliability With pass^k, Not One Green Run

An agent that passes a task once has shown it can do the task. It hasn’t shown it will, and your users run it hundreds of times. This is for teams that already have an agent eval and want it to answer the question that matters in production: how often does the agent get the same task right every time?

One green run measures capability, not consistency

Most agent evals run each task once and report the share that passed. That number is pass@1, and it quietly assumes the run you got is typical. Agents are not deterministic: small differences early in a trajectory cascade into different tool calls, different plans and different outcomes. A study of 60,000 SWE-Bench-Verified trajectories found single-run scores varying by 2.2 to 6.0 percentage points depending on which run was picked, with standard deviations above 1.5 points even at temperature zero (Bjarnason et al. (opens in a new tab)). The same paper warns that reported improvements of 2 to 3 points may be evaluation noise.

So a single run can’t tell a flaky task from a solid one. Two metrics built on repeated trials can:

• pass@k is the chance that at least one of k trials succeeds. It rises with k.
• pass^k (“pass hat k”) is the chance that all k trials succeed. It falls with k.

At k = 1 both equal the per-trial success rate. As k grows they pull apart, and Anthropic’s guidance on agent evals puts the choice plainly: “pass@k for tools where one success matters, pass^k for agents where consistency is essential” (Anthropic (opens in a new tab)). pass^k comes from τ-bench, where GPT-4o-based function-calling agents succeeded on fewer than half the tasks and pass^8 in the retail domain was below 25% (Yao et al. (opens in a new tab)).

The arithmetic is unforgiving. If trials were independent with the same success rate p, pass^k would be p^k:

Real tasks aren’t uniform, which is exactly why you measure per task instead of raising the average to a power. An eval where half the tasks pass every time and half pass 50% of the time has the same mean as one where every task passes 75% of the time, and very different pass^k. IBM Research reported this shape on AppWorld: a ReAct agent averaged 77.4% across five runs, but only 53.0% of tasks passed all five (IBM Research (opens in a new tab)). That’s one team’s result on its own setup, but it’s the gap you should expect to find.

Turn your existing eval into a consistency eval

You don’t need a new eval. Keep the tasks and graders you have, run each task n times, and record every outcome per task rather than one aggregate. Pick n at least as large as the biggest k you want to report; τ-bench used at least three trials per task for its main results (τ-bench paper (opens in a new tab)).

Then estimate both metrics from the per-task counts instead of the naive formulas. With n trials and c passes on a task, the unbiased pass@k estimator from the Codex paper is 1 − C(n−c, k)/C(n, k); the paper shows that computing 1 − (1 − p̂)^k from the observed rate is biased (Chen et al. (opens in a new tab)). τ-bench’s pass^k is C(c, k)/C(n, k), averaged across tasks. Both expand into short products, which avoids huge factorials:

export type TrialOutcome = "pass" | "fail" | "infra_error";

export interface TaskTrials {
  taskId: string;
  outcomes: TrialOutcome[];
}

/** pass^k for one task: C(c, k) / C(n, k), the chance that k trials drawn from n all passed. */
export function passHatK(n: number, c: number, k: number): number {
  if (k > n) throw new Error(`need at least ${k} graded trials, got ${n}`);
  if (c < k) return 0;
  let p = 1;
  for (let i = 0; i < k; i++) p *= (c - i) / (n - i);
  return p;
}

/** Unbiased pass@k for one task: 1 - C(n - c, k) / C(n, k) (Chen et al., 2021). */
export function passAtK(n: number, c: number, k: number): number {
  if (k > n) throw new Error(`need at least ${k} graded trials, got ${n}`);
  if (n - c < k) return 1;
  let q = 1;
  for (let i = 0; i < k; i++) q *= (n - c - i) / (n - i);
  return 1 - q;
}

export function summarize(tasks: TaskTrials[], k: number) {
  let meanRate = 0, atK = 0, hatK = 0, scored = 0, infra = 0, trials = 0;
  const flaky: string[] = [];
  const needMoreTrials: string[] = [];

  for (const { taskId, outcomes } of tasks) {
    trials += outcomes.length;
    infra += outcomes.filter((o) => o === "infra_error").length;
    // Infrastructure failures say nothing about the agent: leave them out of the score.
    const graded = outcomes.filter((o) => o !== "infra_error");
    const n = graded.length;
    const c = graded.filter((o) => o === "pass").length;
    if (n < k) {
      needMoreTrials.push(taskId);
      continue;
    }
    scored++;
    meanRate += c / n;
    atK += passAtK(n, c, k);
    hatK += passHatK(n, c, k);
    if (c > 0 && c < n) flaky.push(taskId);
  }

  const avg = (x: number) => (scored ? x / scored : 0);
  return {
    k,
    tasksScored: scored,
    meanPassRate: avg(meanRate),
    passAtK: avg(atK),
    passHatK: avg(hatK),
    flaky,
    needMoreTrials,
    infraErrorRate: trials ? infra / trials : 0,
  };
}

Report the mean, pass@k and pass^k together, and put the list of flaky tasks at the top of the report. The gap between the mean and pass^k is the headline; the flaky list is where the work is. A task that passes three runs out of five is telling you something specific: an ambiguous instruction, a tool that sometimes returns a different shape, a step where the agent sometimes skips verification. Read those transcripts before touching the prompt.

Budget for it. Every trial is a full agent run, so n trials cost roughly n times a single pass through the eval, in tokens and in wall-clock time. A practical compromise is to run the full n on the smaller set of tasks where consistency matters, rather than multiplying the entire suite.

Separate agent noise from grader and infrastructure noise

A consistency eval measures everything that varies between runs, not only the agent. Two other sources routinely get blamed on the model.

The grader

Code-based graders are fast, cheap and reproducible, but brittle: they fail valid answers that don’t match the expected shape. Model-based graders handle that variation but are themselves non-deterministic and need calibrating against human grading (Anthropic (opens in a new tab)). If an LLM judge scores the same transcript differently on two passes, some of your “flaky” tasks are flaky grading.

Check this directly: take a sample of stored transcripts and grade each one several times without rerunning the agent. Any transcript that gets both a pass and a fail is grader noise, and it has to be fixed (tighter rubric, a code check for the objective part, a human label for the ambiguous cases) before the agent’s pass^k means anything. Then do what Anthropic recommends and read transcripts alongside their grades; a failure should look like a real agent mistake, not an eval quirk.

The environment

Agentic evals run in real environments that install dependencies, run tests and hit APIs, so the infrastructure is part of what you measure. Anthropic found a 6 percentage point gap on Terminal-Bench 2.0 between the most- and least-resourced setups, and an infrastructure error rate of 5.8% under strict resource limits versus 0.5% uncapped (Anthropic (opens in a new tab)). Their recommendation is a guaranteed resource floor plus a separate hard kill ceiling per task, instead of one pinned limit.

That’s why the code above records infra_error as its own outcome. A trial killed by an out-of-memory limit or a runner timeout should be retried, not counted as a fail, and the infrastructure error rate should be reported next to the score so nobody quietly drops it. Keep the environment isolated and reset between trials, too; state that leaks from one trial into the next makes later trials look better or worse than they are. If your agent runs in containers, the isolation setup in Docker Sandboxes for AI Agents is a reasonable baseline.

Decide which number gates a release

Pick the metric by what a failure costs. For a tool where a person reruns a bad answer and moves on, such as a draft generator or a search helper, pass@k or the mean is the honest measure of usefulness. For an agent that acts on its own, or that a user relies on to behave the same way every time (a support agent issuing refunds, an agent that edits files), pass^k is much closer to what users actually experience.

Anthropic’s split between capability and regression evals maps onto this cleanly. Capability evals should start at a low pass rate and show progress; track pass@k and the mean there. Regression evals should sit near 100%; gate on pass^k there, because a regression suite that passes “most of the time” isn’t protecting anything.

Choose k from how often the behavior repeats in production, not from a benchmark convention. The research sources use different values (3 trials, 5, 8), and none of them makes a specific k a standard. A reasonable starting point is the number of times a typical user will hit the same kind of task in a session or a week. The same thinking applies to CI: How to Gate Merges When an Agent Wrote Part of the Diff uses repeat runs and a flake budget for the same reason.

Be careful comparing two configurations. When single runs move by several points on their own, a 2 to 3 point improvement from one run each is not evidence. Run both configurations with the same n on the same tasks, compare per-task pass counts, and size n with a power calculation for the difference you care about, as the randomness study recommends (Bjarnason et al. (opens in a new tab)). If you’re choosing between models rather than prompts, the task-selection advice in Pick a Coding Model With a 20-Task Eval on Your Own Repo still applies; add repeated trials to it before trusting close results.

A few edge cases are worth knowing. Setting temperature to zero doesn’t remove variance, as the SWE-Bench-Verified study measured. Multi-turn evals with a simulated user add the simulator’s randomness to the agent’s. And tasks with several valid answers need graders that accept all of them, or pass^k will punish the agent for being creative.

The rule to adopt: run every regression task at least k times, retry infrastructure failures instead of scoring them, grade a sample of transcripts twice to confirm the grader is stable, and ship only when pass^k on the regression set clears the bar you’d accept from a user’s k-th attempt.

Sources

• Demystifying evals for AI agents (Anthropic) (opens in a new tab)
• Quantifying infrastructure noise in agentic coding evals (Anthropic) (opens in a new tab)
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al.) (opens in a new tab)
• τ-bench paper, full text (opens in a new tab)
• Evaluating Large Language Models Trained on Code (Chen et al.) (opens in a new tab)
• On Randomness in Agentic Evals (Bjarnason, Silva, Monperrus) (opens in a new tab)
• Your Agent Aced the Task. Will It Do It Again? (IBM Research, Hugging Face blog) (opens in a new tab)