Retry Defaults Turn Transient Agent Failures Into Permanent Bad State
Most agent frameworks retry failed tool calls by default. That's the right instinct for GET requests and pure computation. Apply the same default to writes, and every transient bug in your automation becomes a persistent one — quietly, without an error the agent can see.
Where the retry default came from
Retrying is a well-earned default in distributed systems. Amazon's Builders' Library has been documenting this pattern for a decade: transient faults are common, most resolve on a second attempt, and if the API is idempotent the retry is free. AWS SDKs even generate a client token per call so the second attempt lands in the same idempotent session as the first.
Agent frameworks inherited "retry on failure" from that world. LangChain, LlamaIndex, and every home-grown orchestration script I've read defaults tool calls to try-catch-retry with exponential backoff. On the surface it looks like production-grade defensiveness.
The problem is inheriting only the retry, without the idempotency contract underneath.
Three failure modes retry loops make worse
Idempotency mismatch on write-shaped tools. An agent calls create_ticket(...). The tool returns a 504 gateway timeout. The framework retries. The tool succeeded the first time (the timeout was on the response path), so now there are two tickets claiming the same fix, and the audit trail shows one call. Nothing in the agent's context knows this happened. The user sees duplicate work and asks why.
Every write-shaped tool in an agent's toolbox has this problem unless it accepts a caller-supplied idempotency key and the framework passes one through. In my experience most don't. The Model Context Protocol spec doesn't require it. The tools built on top of MCP rarely add it.
Partial-success confusion on batch tools. A tool advertises "write these five records" and either returns a boolean or throws. Under the hood four writes succeed and one fails on a validation error. The tool throws. The framework retries. On the second call, four writes are duplicated (some may fail on a unique constraint, some may not) and one still fails. Now the agent has two problems: the original validation error, and a data set that reflects neither the first attempt nor the second.
The fix at the tool layer is per-record status in the response. The fix at the framework layer is refusing to auto-retry any call that mutates state without either an idempotency key or an explicit "this is a full-batch failure" signal. Neither fix is the default.
Coordination retries in multi-agent chains. Agent A calls Agent B, which calls a tool. The tool fails transiently. B retries and succeeds. But the retry took long enough that A already timed out and re-issued the whole subtask to a fresh instance of B. Two Bs, two tool calls, two writes. This one is hard to spot because the retry lives inside a service you didn't write, and the timeout that triggered A's re-issue looked to A like a legitimate failure.
Any place a retry lives inside a call that itself is retried by its caller, you have this class of bug.
Early fail and dig
The alternative default is what distributed-systems teams have been doing forever. For state-changing calls, fail early and fail loudly. Don't retry inside the tool. Don't retry inside the framework. Push the retry decision up to whoever knows whether re-issuing the call is safe, which is usually the caller who owns the idempotency key.
The rule I use: a tool call is safely retryable only if the tool accepts an idempotency key and honors it, or the operation is provably read-only. Everything else halts on the first error and surfaces the failure to the orchestrator with enough context (request payload, partial state, error class) to decide.
This costs latency on transient failures. That's the trade. What you get in exchange is that when an agent hits an error, the state of the world is knowable, and the fix is a targeted retry with a fresh idempotency key instead of a scavenger hunt through logs asking which of six side effects actually happened.
Why metronix-memory does it this way
Agent memory is a write-heavy surface. Every memory update, every recall log, every attribution to a source is a write. If the memory store retried silently, an agent that made one decision would leave two records of making it, and the next time that agent asked "have I done this before" the answer would be wrong for reasons the agent couldn't see.
MM's write path treats each mutation as compare-and-set against a version. A retry that doesn't carry the expected version fails and returns the current state. The caller decides whether to reconcile and retry with the new version or halt. There's no framework-level auto-retry on the write path. The failure surfaces immediately, and the caller owns the reconciliation.
For contributors, the memory write path is the most concrete place to see this. If you're adding a new memory type or a new source connector, the tools you write are expected to plug into this pattern: accept an expected version, fail loudly if it doesn't match, don't paper over conflicts. The codebase enforces it structurally rather than in a policy doc, and the "how to add a memory type" walkthrough in the repo is where the pattern gets demonstrated on real code.
What I don't have a clean answer for
For tools you don't own (third-party APIs, other teams' services) you can't add an idempotency-key contract retroactively. The practical fallback is to record every outbound call in a durable log before it fires and reconcile on failure, but reconciliation is code you have to write per tool.
If you're running production agents against a lot of third-party tools that don't offer idempotency keys, what's your actual reconciliation strategy? Especially curious about anyone using deterministic replay of the tool-call log versus writing per-tool "did this happen" probes.