Durable Execution for AI Agents: Surviving a Crash at Step 40

An agent is forty steps into a fifty-step task. It has read thirty files, made twenty-five model calls, opened a pull request and posted a summary to Slack. Then the process restarts for a routine deploy. Without durable execution there are two outcomes, both bad: the run starts over, paying for every model call again and posting to Slack a second time, or it’s lost. This is for teams whose agent runs last longer than a single request.

Why “just retry it” breaks for agents

Retrying a whole request is fine when the request takes two seconds and changes nothing. Agent runs violate both conditions.

Model calls are expensive, and they aren’t repeatable. Rerunning an agent from the start doesn’t reproduce the original run; it produces a different one, possibly with a different plan and different tool calls, and you pay for all of it again. The forty steps you lost weren’t a cached computation. They were a particular path through the task that you won’t get back.

Side effects repeat. A naive retry re-sends the email, re-comments on the ticket, opens a second pull request. Teams usually discover this from a customer who got the same message twice.

And agents wait. A run that needs a human to approve a change might sit for hours. Holding a process open for that is fragile: every deploy, scale-down or crash during the wait loses the run.

How durable execution works

The idea is the same across the platforms that offer it: record the result of every step, and when a run resumes, replay the orchestration code from the top while completed steps return their recorded results instead of executing again. AWS’s description of Lambda durable functions puts it plainly: “your code runs from the beginning but skips over completed checkpoints, using stored results instead of re-executing completed operations” (AWS Lambda durable functions (opens in a new tab)).

That mechanism solves the non-determinism problem neatly. Once a model call is a step, its answer is a recorded fact. Cloudflare’s guide to durable agents makes each model call and each tool call its own step, so that after a crash “The response is persisted. On resume, it skips the LLM call and moves to tool execution” (Build a Durable AI Agent (opens in a new tab)). Replay follows the same path the run was already on, because the decisions that shaped it are read back rather than made again.

Waiting changes too. Instead of a process sitting idle, the run is suspended and its state stored until a timer fires or an event arrives. On Cloudflare Workflows, instances that are sleeping or waiting for an event “do not count towards concurrency limits”, and a sleep can last up to a year (Workflows limits (opens in a new tab)). AWS says its waits suspend execution without compute charges, and Vercel describes its sleeps and hooks the same way. An approval that takes three days costs storage, not a server.

The fact that Cloudflare, Vercel and AWS all ship the same model is the useful signal here. Durable execution isn’t a vendor feature to evaluate; it’s a pattern, and the design decisions below apply on any of them.

Where step boundaries go

The platform handles persistence and replay. What it can’t decide is where your steps begin and end, and that’s most of the design work.

Make every model call its own step, and every tool call that changes something outside the run its own step. That’s what makes a resumed run skip exactly the work that finished and redo only what didn’t. Coarser steps are cheaper to record but repeat more on failure: a step that wraps three model calls pays for all three again if the third one fails.

Watch what you’re storing. Step results go into the run’s state, and Cloudflare caps a single step’s return value at 1 MiB, which a tool that reads a large file or a whole web page will hit. Store large outputs in object storage and return a reference. The same records raise a question teams tend to miss: on Vercel, “All inputs and outputs are recorded in an event log” (Workflow Concepts (opens in a new tab)), which for an agent means prompts, model responses and tool results, including any personal data in them. That store needs the same retention and access rules as your other user data.

Keep the orchestration code deterministic. Replay runs that code again, so anything that can differ between runs must come from a step. Cloudflare’s rules for workflows are specific: wrap Math.random() and Date.now() in steps, build state only from step results because “Workflows may hibernate and lose all in-memory state”, and name steps deterministically, since “Step names act as the ‘cache key’” (Rules of Workflows (opens in a new tab)). A timestamp read outside a step takes a different value on replay, and the run quietly takes a different branch.

Idempotency is still your job

Durable execution guarantees that a completed step won’t run again. It can’t guarantee that a step which crashed halfway didn’t partly happen. If a step posts to Slack and the process dies before the result is recorded, the step retries and posts again. Cloudflare’s first rule addresses exactly this: non-idempotent calls should only be made “after checking if the operation is still needed”.

For classic workflows, engineers design idempotency call by call. Agents make that harder, because the model chooses which tools to call at runtime. The workable approach is to make every tool with side effects idempotent at the tool layer: derive an idempotency key from the run and the step, pass it to APIs that accept one, and check for the existing comment, ticket or pull request before creating another. Once that’s in place, retries are boring, which is the goal.

Deploys and long-running runs

A run that starts on Monday and waits for approval until Thursday will see deploys in between. Replaying Monday’s run against Thursday’s orchestration code would be wrong, since the code that made the recorded decisions has changed. Platforms deal with this by pinning runs to the deployment they started on. Vercel’s docs spell out the consequence: “a rollback does not stop runs on the deployment you rolled back from”, and runs pinned to a deleted deployment never complete on their own, so they have to be cancelled.

The practical rule is to treat old deployments as live while long runs are pinned to them. A fix you just shipped won’t reach runs that are already waiting, and the cleanup step after a bad release includes cancelling the runs it started.

When a plain retry is enough

Durable execution has costs: stored state and events (often billed), discipline about determinism, and replays that are harder to debug than a single function call. An agent that finishes in seconds, makes a few calls and changes nothing outside itself doesn’t need it. A timeout and a retry are simpler and correct.

Use durable execution when a run is likely to outlive a deploy, waits on a person or an external event, or causes side effects you can’t safely repeat. Most agents that do real work cross at least one of those lines.

It also helps to be clear about what durability doesn’t give you. It makes a run survive crashes; it doesn’t make it succeed. An agent that completes its task six times in ten is just as unreliable when it’s durable, so measure success separately, as in measuring agent reliability with pass^k. And if you’re deciding between a managed durable service and running a workflow engine yourself, the VPS cost teardown covers the trade-offs of operating that kind of state on your own box.

Before an agent ships, list every tool call that changes something outside the run, and for each one answer what happens if it executes twice. If any answer is “a customer notices”, that tool needs an idempotency key before the agent needs anything else.

Sources

• Lambda durable functions (opens in a new tab), AWS Lambda Developer Guide
• Build a Durable AI Agent (opens in a new tab), Cloudflare Workflows docs
• Limits (opens in a new tab), Cloudflare Workflows docs
• Rules of Workflows (opens in a new tab), Cloudflare Workflows docs
• Workflow Concepts (opens in a new tab), Vercel Docs