Pick a Coding Model With a 20-Task Eval on Your Own Repo

Model choice on most teams is settled by whoever argued last, backed by a leaderboard screenshot and a couple of anecdotes about which assistant “gets” the codebase. That argument is cheap to end: twenty tasks pulled out of your own git history, each with a script that returns zero or non-zero. This is for teams running coding agents in CI or on developer machines who need a defensible answer and a cheap way to change it later.

Why the public leaderboard doesn’t answer your question

SWE-bench Verified is a 500-problem subset of SWE-bench that human engineers confirmed as solvable, produced with OpenAI’s Preparedness team in August 2024 (SWE-bench README (opens in a new tab)). It’s a genuinely good benchmark and its harness design is worth copying. What it can’t tell you is how a model handles your 400-file Next.js app with a hand-rolled ORM layer, three generations of test conventions, and a lint config that rejects half of what the model writes.

The distribution mismatch is the whole problem. Public benchmarks are built from open-source Python repos whose issues and fixes have been on the internet for years. Your internal service has none of that prior exposure, but it also has conventions a model can only discover from the files you put in context. Vendors have started selling evaluations against private enterprise repositories, such as Real-SWE (opens in a new tab), but those are vendor-published numbers on codebases you can’t inspect. Treat them as marketing until someone shows you the harness.

Twenty tasks from your own repo is a small sample and will not give you a statistically tight ranking. It will reliably tell you which candidates fall over on your build system, your test runner, and your file layout, which is usually where the real differences show up.

Mine 20 tasks out of git history

The good tasks are merged commits that changed both source and tests. The test change is what makes the task gradeable: it’s your fail-to-pass evidence, already written by a human who knew the correct behavior.

git log --since="18 months ago" --merges --format="%H %s" \
  | while read -r sha subject; do
      files=$(git show --name-only --format="" "$sha")
      echo "$files" | grep -qE '\.(test|spec)\.(ts|tsx)$' || continue
      echo "$files" | grep -qvE '\.(test|spec)\.(ts|tsx)$' || continue
      count=$(echo "$files" | grep -c .)
      [ "$count" -le 12 ] && echo "$sha  ($count files)  $subject"
    done

That gives you a few hundred candidates. Read them and pick twenty by hand. The selection matters more than the tooling, so aim for a spread rather than twenty tidy bug fixes:

• eight to ten regression fixes where the test change is a clear behavioral assertion;
• three or four cross-cutting migrations, the kind that touch many files but follow a mechanical rule (a client/server boundary change in an App Router codebase is a good one, since the rules about where use client belongs are exactly the sort of convention a model either knows or invents);
• two or three tasks whose obvious fix is wrong, where the correct patch depends on something only visible in a neighboring module;
• one or two underspecified tasks where the right answer is to ask a question or change nothing.

That last category is the one teams skip, and it’s the one that separates models in practice. A model that confidently patches an ambiguous ticket costs you more review time than a model that stops.

Store each task as a record, not as prose. The fields you need are the base commit the agent starts from, the fix commit you’re hiding, the test files that define success, and the prompt the agent actually receives.

{
  "id": "0007-stale-cursor-pagination",
  "base_commit": "9f2c1ab",
  "fix_commit": "4d81e77",
  "prompt": "Listing the second page of activity events returns items already shown on page one when two events share a created_at timestamp. Fix it.",
  "test_files": ["src/server/activity/pagination.test.ts"],
  "fail_to_pass": ["src/server/activity/pagination.test.ts"],
  "pass_to_pass": ["src/server/activity/", "src/server/db/"],
  "golden_diff_lines": 31
}

Write the prompt the way a ticket arrives, not the way the fix reads. If you paste in the commit message from the real fix, you’ve leaked the answer and you’re now benchmarking paraphrasing.

The grader is the hard part

A task is only useful if a script decides it. The structure worth copying from SWE-bench is the split between fail-to-pass tests, which must go from failing to passing, and pass-to-pass tests, which must not regress (SWE-bench README (opens in a new tab)). Without the second half, a model that deletes an inconvenient assertion scores the same as one that fixes the bug.

The non-obvious defense: restore the test files from the fix commit after applying the agent’s patch. The agent never sees them and cannot edit its own grader.

set -euo pipefail

task="$1"; patch="$2"; repo="${REPO:-$PWD}"
base=$(jq -r .base_commit "$task")
fix=$(jq -r .fix_commit "$task")

git -C "$repo" checkout -q --detach "$base"
git -C "$repo" clean -qfd

if ! git -C "$repo" apply --index "$patch"; then
  echo "apply_failed"; exit 2
fi

# Grader wins any conflict with the agent: tests come from the fix commit.
jq -r '.test_files[]' "$task" | xargs git -C "$repo" checkout "$fix" --

if ! pnpm -C "$repo" vitest run $(jq -r '.fail_to_pass[]' "$task"); then
  echo "f2p_failed"; exit 1
fi

if ! pnpm -C "$repo" vitest run $(jq -r '.pass_to_pass[]' "$task"); then
  echo "p2p_regressed"; exit 1
fi

echo "pass"

Exit codes carry information here on purpose. apply_failed is a different problem from p2p_regressed: the first usually means your harness handed back a malformed diff or the model wrote a full-file rewrite, and it tells you more about your agent scaffold than about the model. Keep the two separate in your results or you’ll blame a model for your patch parser.

Run each task from a clean checkout of the base commit. Verify before you start that the fail-to-pass tests genuinely fail at base and the pass-to-pass set genuinely passes. Roughly a third of mined candidates fail this check, usually because the fix depended on a migration or a fixture that landed separately.

Run untrusted agent output in a sandbox

An agent eval executes model-generated code against your repository, with your test runner, on whatever machine you point it at. Your repo’s pretest hook, its lifecycle scripts, and any .env the runner can read are all in scope. That’s the actual security boundary, and it isn’t the model’s alignment.

Tailscale’s Aperture router puts agent execution in Vercel Sandbox for exactly this reason, grants access by tailnet identity instead of handing out provider API keys, and exposes a zeroDataRetention flag that can be set globally or per request to restrict routing to ZDR-compliant providers (Vercel customer story (opens in a new tab)). SWE-bench takes the container route for the same class of reasons, running its evaluation harness in Docker so results are reproducible and isolated (evaluation guide (opens in a new tab)).

For an internal eval, a disposable container with no network egress except the model endpoint, a read-only mount of your package cache, and no credentials beyond the repo checkout is enough. Don’t run the first version of this on a developer laptop that has production credentials in its keychain.

What to report besides pass rate

Run every task at least three times per model. Coding agents are nondeterministic even at low temperature, and a single 14/20 tells you very little about whether the next run is 11 or 17. Report per-task outcomes, not just the aggregate, because the interesting signal is usually a cluster: one model passes every regression fix and fails every migration.

Three numbers earn their place next to pass rate. Cost per task, since a model that passes 16/20 for a quarter of the price of the one that passes 17/20 is the better default. Wall clock, because an agent that takes twenty minutes per task will not be used interactively regardless of its score. And diff size relative to golden_diff_lines, which catches models that pass tests by rewriting a module.

Context limits show up in this table in a way leaderboards hide. DeepSeek V4.1 Flash, for example, is documented with a 1M-token context window, responses up to 384,000 tokens, and prompt caching, under the model id deepseek/deepseek-v4.1-flash (Vercel changelog, 2026-09-09 (opens in a new tab)). If three of your twenty tasks involve a migration across forty files, that difference is measurable in your results rather than theoretical.

Put the winner behind a router

The point of doing this properly is that you’ll do it again in four months. Wire the result so that redoing it costs a config change, not a refactor.

With Vercel AI Gateway, routing rules rewrite a request from one model to another, or deny a model outright, and they apply to every request made with the team’s Gateway credentials (routing rules docs (opens in a new tab)). That’s the mechanism for “we switched our default coding model” without touching application code. Model fallbacks are a separate feature: a list of backup models tried in order when the primary fails or is unavailable (fallbacks docs (opens in a new tab)). Don’t reach for fallbacks when you mean routing. A fallback silently changes which model handled a request, which is fine for availability and quietly poisonous inside an eval run, where you need to know exactly which model produced each patch.

import { generateText } from "ai";

const CANDIDATES = [
  "deepseek/deepseek-v4.1-flash",
  // add the other candidates you are evaluating
] as const;

export async function proposePatch(
  model: (typeof CANDIDATES)[number],
  prompt: string,
  repoContext: string,
) {
  const { text, usage } = await generateText({
    model, // Gateway resolves the id; routing rules can rewrite it server-side
    system: "Return a unified diff and nothing else. Do not modify test files.",
    prompt: `${repoContext}\n\n---\n\n${prompt}`,
  });

  return { diff: text, usage };
}

The model id being a plain string is the design goal. Your eval harness iterates over candidates; production reads one value from config; the Gateway can override both centrally.

Two practical notes. Gateway doesn’t mark up provider token pricing and charges no platform fee on inference, including BYOK requests, so your eval cost numbers stay comparable to going direct (changelog (opens in a new tab)). And if your team uses Claude Code, Codex, or Cursor, those can be pointed at Gateway models with npm i -g vercel@latest followed by vercel ai-gateway setup (coding agents guide (opens in a new tab)), which means the model your developers use interactively can be the model your eval actually scored. If you’d rather not depend on a single vendor, a self-hosted LiteLLM proxy or OpenRouter gives you the same indirection; verify the routing and fallback semantics in their current docs before assuming they behave like Gateway’s.

Edge cases worth knowing about

Flaky tests will eat this project if you don’t quarantine them first: run your pass-to-pass sets five times at base and drop anything that isn’t stable. Monorepos need the task record to pin a workspace, or the pass-to-pass run takes an hour. Tasks that depend on a schema migration need the migration applied in the base image rather than expected from the agent. And if your work is UI-heavy, note that SWE-bench’s own multimodal test split is evaluated privately through sb-cli rather than the local harness (SWE-bench README (opens in a new tab)) — screenshot-based grading is a harder problem and probably not where your first twenty tasks should go.

One more: you are benchmarking the model and your agent scaffold together. If you change the retrieval strategy, the system prompt, or the patch format, the old scores are no longer comparable. Version the harness alongside the tasks.

When to re-run it

Set three triggers and automate the first one. Re-run the full suite when a candidate model ships a new version, when you change the agent scaffold, and on a fixed quarterly cadence regardless. Retire any task that all candidates pass three times running, because it has stopped discriminating, and replace it from the same mining script against the last quarter of merged commits. Keep the suite at twenty. The moment it grows to sixty, nobody runs it, and you’re back to arguing from screenshots.

Sources

• SWE-bench (GitHub README) (opens in a new tab)
• SWE-bench evaluation guide (opens in a new tab)
• Vercel AI Gateway: routing rules (opens in a new tab)
• Vercel AI Gateway: model fallbacks (opens in a new tab)
• Vercel AI Gateway: coding agents (opens in a new tab)
• DeepSeek V4.1 Flash now available on AI Gateway (changelog) (opens in a new tab)
• How Tailscale built a customer-facing model router on AI Gateway (opens in a new tab)
• Real-SWE benchmark (vendor page) (opens in a new tab)