How to Gate Merges When an Agent Wrote Part of the Diff

A PR shows up with a label saying an agent wrote most of it. CI is green on the first attempt, review is a skim because the tests pass, and a day later the checkout flow is quietly dropping one order in twenty. Nothing in the pipeline lied; the pipeline just answered a narrower question than anyone thought it was answering.

There are two different sources of nondeterminism hiding in that sentence, and they need separate treatment. One is nondeterminism at write time: an agent produced the diff, and it could have produced a different diff. The other is nondeterminism at execution time: a model call, a browser-driving agent, or a tool loop runs inside the code path under test, so every run is a fresh sample. The position here is that a single green run is a smoke test, not a release signal, and that closing the gap takes three things: assertions on observable state, repeated runs, and a flake budget your team defines and enforces.

Why one run proves less than it used to

Be honest about the first case. If an agent wrote plain deterministic code — a reducer, a migration, a route handler with no model calls in it — then running the suite fifty times mostly measures your test suite’s flakiness, not the agent’s. The diff is fixed once it’s committed. Repeating the run is cheap insurance against ordering and timing bugs, and it’s worth doing on a nightly job, but it isn’t the main event.

The argument gets much stronger the moment something nondeterministic executes inside the test path. An agent that drives a browser, a retrieval step, a tool-calling loop, an LLM that classifies an input before your code branches on it: each of those turns your test into a sampled measurement. One pass tells you the happy path is reachable. It tells you nothing about the distribution. As one practitioner puts it, almost any agent change can be made to look good with one carefully chosen prompt (opens in a new tab).

There’s a second-order problem that’s easy to miss in review. Agent-authored diffs tend to arrive with agent-authored tests. A green run can mean the assertions were written to describe whatever the implementation happens to do, which is a tautology with a checkmark next to it. That’s the shape of the verification bottleneck (opens in a new tab): generation got fast, and deciding whether the output is correct did not.

Assert on observable state, not the tool’s success return

A tool call returning success means the call completed. It does not mean the effect you wanted happened. The canonical version of this failure is an agent clicking Checkout, the browser tool returning success, and no order existing anywhere (opens in a new tab). The run completes, the reporter is green, and the only thing that was verified is that a function returned.

Substitute assertions whose subject is state the system produced:

• a database row in the expected status, fetched by a key the test controls;
• a message on the queue, or a consumer offset that advanced;
• an outbound HTTP call recorded by your mock server, with the body you expect;
• a rendered element containing the real order id, not a spinner that resolved;
• a log line carrying the correlation id from this run.

import { test, expect } from '@playwright/test';
import { resetCart, findOrderByCartId } from './helpers/fixtures';
import { runAgentStep } from './helpers/agent';

test('agent checkout leaves a paid order behind', async ({ page }, testInfo) => {
  const cartId = `cart-${testInfo.parallelIndex}-${testInfo.retry}`;
  if (testInfo.retry) await resetCart(cartId);

  await page.goto(`/cart/${cartId}`);
  await runAgentStep(page, 'complete the checkout'); // return value deliberately ignored

  await expect(page.getByTestId('order-confirmation')).toBeVisible();
  const order = await findOrderByCartId(cartId);
  expect(order?.status).toBe('paid');
});

That example assumes your test process can reach the application database directly, which is true for most single-service setups and false for most multi-tenant staging environments; if it’s false, assert through an internal read API instead and make sure the API isn’t served from the same in-memory cache the write path just populated.

If you change nothing else this quarter, delete every assertion whose subject is a tool’s own return value. It’s a grep, it’s a small PR, and it removes the class of green run that means nothing.

Run it N times, then look at the spread

The simplest way to execute the same job N times is a matrix over a run-index array. Each entry is an independent job with its own runner, so state doesn’t bleed between samples the way it does when you loop inside a single job.

name: agent-diff-gate
on: pull_request

jobs:
  repeat:
    if: contains(github.event.pull_request.labels.*.name, 'agent-authored')
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        run-index: [1, 2, 3, 4, 5]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Run suite (sample ${{ matrix.run-index }})
        continue-on-error: true
        env:
          PLAYWRIGHT_JSON_OUTPUT_NAME: results/${{ matrix.run-index }}.json
        run: npx playwright test --reporter=json
      - uses: actions/upload-artifact@v4
        with:
          name: results-${{ matrix.run-index }}
          path: results/

continue-on-error: true is load-bearing here. A failing sample is data, not a reason to stop collecting the other samples, and the aggregation job downstream needs all of them. Artifact names have to be unique per matrix leg.

Be precise about what Playwright does for you natively. Retries are off unless you turn them on: “By default failing tests are not retried” (opens in a new tab), and you enable them with retries: N in the config or --retries=3 on the CLI. When retries are on, Playwright classifies results as passed (green on the first run), flaky (failed first, passed on a retry), and failed (failed every attempt). That classification is genuinely useful, and it is not the same thing as N independent samples: a retry only happens after a failure, so a suite with retries enabled produces a biased sample of the runs you care about.

Retries also need help staying independent. The same doc shows using TestInfo.retry to clear server-side state before the next attempt, which is the tell: a second attempt against a dirty database is not a fresh sample of anything. Playwright discards the worker process and browser after a failure, so browser-side state is clean, but rows your test wrote are still sitting there.

On N: no first-party documentation prescribes a number, and anyone who gives you one without measuring your suite is guessing. Start with something small enough to run on every agent-labelled PR — five is a common starting point because it’s one matrix line and roughly one runner’s worth of parallel cost — and a larger N, twenty or more, on the nightly or pre-promotion job. Then tune based on what your own pass-rate distribution looks like. The number is a trade against CI minutes and nothing else.

Retries and reruns are labels, not gates

GitHub Actions lets you re-run a whole workflow run, all failed jobs, or specific jobs for up to 30 days after the initial run (opens in a new tab). That’s a useful feature and it is also exactly the mechanism that becomes “rerun until green” on a Friday afternoon. Playwright’s flaky classification records that a test needed a retry; it doesn’t block anything. Neither product ships a promotion gate, and neither claims to.

So treat every rerun as data. Record attempt counts per PR and surface them in the merge check, rather than letting the last attempt overwrite the story of the previous four.

A flake budget you can actually enforce

“Flake budget” is team policy, not a vendor feature. No GitHub Actions, GitLab CI, CircleCI or Playwright documentation defines one, which means you own the definition, the threshold and the enforcement. That’s the point of the exercise.

The shape that works: pass rate over N runs, a threshold that differs by target branch, and an automatic block when the rate falls below it. Keep merge-to-main and promote-to-production separate. A 95% pass rate over five samples might be fine to merge behind a flag and clearly not fine to promote, and collapsing those into one number forces you to pick the wrong threshold for one of them.

Store the raw material as a JSON artifact per sample, then have one aggregation job download all of them, compute the rate, and post a required status check. The fields you need are small:

{
  "commit": "9f2c1ab",
  "runIndex": 3,
  "agentAuthored": true,
  "results": [
    { "testId": "agent-checkout:paid-order", "attempt": 0, "outcome": "failed" },
    { "testId": "agent-checkout:paid-order", "attempt": 1, "outcome": "passed" }
  ]
}

Keeping attempt separate from outcome is what lets you distinguish “passed” from “passed eventually” months later, when someone asks whether this test has always been like this.

One trade-off to name before you turn it on: a strict budget will immediately stop the pipeline on tests that were flaky long before any agent touched the repo. That backlog is real work. Quarantine those tests into a non-blocking suite with an owner and a date, and pay them down. Permanent exemptions turn the budget into decoration.

Who decides what correct means

N runs verify the assertions somebody wrote, and nothing beyond them. If the contract is undefined, repeated agreement is repeated agreement about nothing in particular — which is the contract discovery problem (opens in a new tab) rather than a testing problem.

The practical answer is ordering. Write the contract first: expected state after the operation, the error cases and what each one leaves behind, whether the operation is idempotent and what a duplicate call should do. Then let the agent write the implementation against it. Reviewers read the contract carefully and the diff quickly, instead of the reverse. The same instinct applies one level up, when you’re choosing which model to let near the repo at all: run a 20-task eval on your own codebase before it starts opening PRs.

Wiring it into the pipeline

Roll it out in this order, because each step makes the next one cheaper:

1. Label agent-authored PRs, automatically if your tooling can set it, manually if not. Everything else keys off this label.
2. Delete assertions on tool return values and replace them with state assertions. Do this before adding repeated runs, or you’ll pay five times over to verify nothing.
3. Add the N-run matrix behind the label, with fail-fast: false and per-sample artifacts.
4. Add the aggregation job that computes pass rate and posts a required status check. Report only, no blocking, for two weeks.
5. Read the two weeks of data, set the threshold where it would have caught the failures you actually had, and turn on blocking for promotion first, merge second.
6. Quarantine the pre-existing flaky tests the new check surfaces, with owners and dates.

If you want a single rule to take away: never let a test assert on a return value from something that could have been mocked, and never let a green run count as a signal unless you know how many attempts it took.

Sources

• Playwright test retries documentation (opens in a new tab)
• GitHub Docs: Re-run workflows and jobs (opens in a new tab)
• One Passing Agent Run Is Not a Release Signal (opens in a new tab)
• My Agent Returned Success. The Browser State Said Otherwise (opens in a new tab)
• The Verification Bottleneck in AI-Generated Software (opens in a new tab)
• The Contract Discovery Bottleneck (opens in a new tab)