Tag · 3 stories
A model upgrade changes behaviour your prompts depend on, and the failures return HTTP 200: a missing field, a tool that isn’t called, a refusal, a bill that grows. How to build a small regression suite from your own traffic and decide from the diff.
A single passing run shows an LLM agent can do a task, not that it will every time. How to run repeated trials, estimate pass@k and pass^k per task, separate grader and infrastructure noise, and pick the number that gates a release.
Public leaderboards measure someone else’s repo. Here’s how to build a 20-task eval from your own git history, grade it with deterministic pass/fail scripts in a sandbox, and put the winner behind a router so the next switch is a config change.
