Review an AI-Written Pull Request in This Order
Green CI only proves the branch builds and passes the tests it ships with. Read the task, the file list and the test diff before the implementation, and run the new tests against the old code.
testing
Tag · 3 stories
Stories tagged testing
Measure Agent Reliability With pass^k, Not One Green Run
A single passing run shows an LLM agent can do a task, not that it will every time. How to run repeated trials, estimate pass@k and pass^k per task, separate grader and infrastructure noise, and pick the number that gates a release.
How to Gate Merges When an Agent Wrote Part of the Diff
A single green CI run tells you a change can pass, not that it does pass. Three gates that hold up when part of the diff was written by an agent: assert on observable state, repeat the run N times, and enforce a flake budget you define yourself.