Idempotency Explained: Keys, Safe Retries & Exactly-Once Effects

A payment request times out. The client does not know whether the charge went through, and it has to decide something: retry, and risk charging the customer twice, or give up, and risk a customer who was charged but owns nothing. Idempotency is how you make that decision boring — retry, always, because the second request is guaranteed to change nothing.

That guarantee is not what HTTP gives you for free, and it is not a line you can add to a design at the end. This post covers the arithmetic that makes retries mandatory, what an idempotency key actually has to store, the concurrency bug that almost every first implementation ships, how long to keep keys and what the storage costs, what changes when the effect lives outside your database, and the four problems idempotency is routinely expected to solve and does not.

The arithmetic that makes retries mandatory

Take an API serving 20 million write requests a month. Suppose 0.05% of them end ambiguously — a client-side timeout, a connection reset, a 504 from a proxy that had already forwarded the request upstream. That is 10,000 requests a month where the caller has no idea what happened.

You cannot engineer that number to zero. The acknowledgement is itself a message that can be lost, which is the two generals problem, and no amount of reliability work removes it. So those 10,000 requests have to be resolved by policy, and there are only two policies:

• Never retry. The successful half of those requests are fine; the failed half are silently dropped. You have built a payment API that loses about 5,000 payments a month and tells nobody.
• Always retry. Nothing is dropped, but the half that already succeeded get applied a second time. Roughly 5,000 duplicate charges a month, each one a support ticket and a chargeback.

Neither is acceptable, and the choice is a false one. Make the operation idempotent and "always retry" costs nothing: the duplicate half becomes 5,000 harmless no-ops. That is the entire value proposition, and it is worth stating in exactly those terms in an interview, because it frames idempotency as a reliability requirement rather than an API nicety.

Idempotent is not the same as safe

An operation is idempotent when applying it more than once has the same effect as applying it once. It is safe when it has no effect at all. Safe implies idempotent; the reverse is not true, and the distinction is where most confusion lives.

The HTTP spec assigns these properties to methods, but it is describing a contract you are expected to honour, not a behaviour the protocol enforces. Your handler decides.

Method	Safe	Idempotent per spec	What that means in practice
GET	Yes	Yes	True unless your handler writes something — a view counter makes GET neither
PUT	No	Yes	True when the body is absolute state ({"status": "shipped"}), false when it is a delta
DELETE	No	Yes	True — the second call is a no-op. Return 204 both times; a 404 on the retry breaks the contract
POST	No	No	The method that creates things. This is where keys are needed
PATCH	No	No	{"status": "shipped"} is idempotent; {"$inc": {"qty": 1}} is not

The practical reading: the only reason PUT is idempotent and POST is not is that PUT names the resource and POST asks the server to invent one. If you can design the write as "put this state at this address," you get idempotency structurally and need none of the machinery below. Most of the time you cannot, because the address is a database-assigned id.

What the client can actually see

The reason you need a key at all is that the two interesting failures are indistinguishable from outside.

A common instinct is to resolve the ambiguity by querying: time out, then GET /charges?order=123 to see whether it exists. That fails for three reasons. The write may still be in flight, so absence proves nothing. The read may hit a replica that has not caught up, which is a replication lag problem you have now coupled your correctness to. And it does not generalise — plenty of operations leave nothing queryable behind. Deduplicating on the write path is strictly simpler than reconstructing state on the read path.

The key is a statement of intent

An idempotency key is a client-generated opaque string — a UUIDv4 is fine — that identifies one intent, not one attempt. The client mints it when the user presses the button and reuses it for every retry of that same action.

That sentence contains the single most common bug in the whole topic. If the key is generated inside the retry loop, every attempt carries a fresh key, the server sees three distinct intents, and you have built an elaborate dedupe layer that deduplicates nothing. Mint outside the loop, persist it with the pending request if the client has local storage, and treat it as part of the request's identity.

Two scoping decisions are worth making explicitly:

Scope the key to the caller. The uniqueness constraint should be on (account_id, key), not key alone. Keys are client-chosen, and a client that ships "1" must not be able to collide with — or read the stored response of — another tenant.

Scope it to the operation. The same key arriving at a different endpoint is a client bug, not a duplicate. Either namespace the stored key by route, or treat the mismatch as an error. Silently replaying a charge response to a refund request is worse than failing.

When the effect is local, one transaction is the whole mechanism

Here is the part that gets over-engineered constantly. If the operation's entire effect is rows in your own database, you do not need a state machine, a lock service, or a background reaper for correctness. You need a unique constraint and one transaction.

create table idempotency_keys (
  account_id    bigint      not null,
  key           text        not null,
  request_hash  bytea       not null,
  status_code   int,
  response_body jsonb,
  created_at    timestamptz not null default now(),
  primary key (account_id, key)
);

def create_charge(account_id, key, body):
    fp = sha256(canonical_json(body))

    with db.transaction() as tx:
        tx.execute("set local lock_timeout = '3s'")
        claimed = tx.execute(
            """insert into idempotency_keys (account_id, key, request_hash)
               values (%s, %s, %s)
               on conflict (account_id, key) do nothing
               returning key""",
            (account_id, key, fp),
        ).fetchone()

        if claimed is None:                      # another attempt owns this key
            prior = tx.execute(
                """select request_hash, status_code, response_body
                     from idempotency_keys
                    where (account_id, key) = (%s, %s)""",
                (account_id, key),
            ).fetchone()
            if prior.request_hash != fp:
                raise HttpError(422, "idempotency key reused with a different body")
            return prior.status_code, prior.response_body, {"Idempotent-Replay": "true"}

        charge = insert_charge(tx, account_id, body)        # the actual effect
        tx.execute(
            """update idempotency_keys
                  set status_code = 201, response_body = %s
                where (account_id, key) = (%s, %s)""",
            (charge.as_json(), account_id, key),
        )

    return 201, charge.as_json(), {}

Four details carry all the weight.

The insert is the lock. on conflict do nothing does not return immediately when a concurrent transaction holds an uncommitted row for that key — Postgres makes the second inserter wait on the first transaction. That wait is the serialization point, and you get it from the index you already needed.

A rolled-back attempt frees the key. If the first transaction aborts, the waiting insert proceeds and succeeds. You do not need to distinguish "in progress" from "failed and should be retryable," because the transaction boundary does it. This is the main reason to prefer insert-first over any scheme with an explicit state column.

lock_timeout is not optional. Without it, a slow first attempt makes every retry hang for as long as it runs. Three seconds, then return 409 and let the client back off.

The dedupe row and the effect commit together. There is never a moment where the system believes it processed something it did not, or vice versa. That atomicity is the thing you are buying, and it is why the unique index belongs in the same database as the effect — a Redis SETNX outside the transaction can succeed while the transaction rolls back, which turns a retryable failure into a permanent one.

The race every first implementation ships

The intuitive implementation is check-then-act: look the key up, and if it is absent, do the work. It is wrong, and it is wrong in the exact window that matters — two retries of the same intent arriving at once, which is what happens when a client times out at 5s and the first attempt is at 5.1s.

The general rule is worth internalising beyond idempotency: do not ask the database a question whose answer you then act on. Perform the write that can only succeed once, and let it tell you whether you were first. The same reasoning is why deleting a cache key beats updating it under concurrency, covered in caching strategies.

Store the response, not just the fact

A dedupe table that records only "this key was used" is half an implementation. The retrying client still needs an answer, and the answer it needs is the one the original attempt produced — the same charge id, the same 201.

So store the status code and the serialized response body, and on replay return them verbatim. Three rules for the edges:

• A completed duplicate returns the original response, not 409. The whole point is that the client does not have to care. Add a response header such as Idempotent-Replay: true so operators can see it in logs, and keep the status code identical.
• An in-flight duplicate returns 409 with a Retry-After. The honest answer is "ask again shortly," and this is the one case where the client must handle something.
• Same key, different body returns 422. Store a hash of the canonicalised request — sorted keys, normalised numbers — and compare on replay. This catches a real class of client bug: a retry loop that reuses the key while the amount changed from 40.00 to 4000. Failing loudly here has saved more money than any other part of the mechanism.

Storing the response also quietly handles non-determinism. If the handler stamps created_at, picks a random coupon, or allocates an id, recomputing on replay would produce a different answer. Replaying bytes cannot.

Retention, and what it costs

Keys accumulate fast, and the retention window is a contract you publish, not an implementation detail.

Say the service takes 500 keyed writes per second. That is 500 × 86,400 = 43.2 million rows a day. A row is a 36-byte key, a 32-byte hash, two timestamps, an integer, and a response body that is realistically 200–400 bytes of JSON; with per-row and index overhead, budget about 500 bytes. So roughly 21 GB a day.

Retention	Rows	Approx. size
24 hours	43M	~21 GB
7 days	302M	~150 GB
30 days	1.3B	~650 GB

Twenty-one gigabytes is a non-event; 650 GB is a second, high-write table whose primary key index no longer fits comfortably in memory, which makes every insert a disk-seeking one — the write-amplification effect described in database indexing. Stripe's keys expire after 24 hours, and that is the right order of magnitude: no sane client retries a day later, and if yours does, it should be creating a new intent rather than resurrecting an old one.

Expire by dropping daily partitions, not with delete from idempotency_keys where created_at < .... A nightly mass delete on your highest-write table generates enormous WAL, bloats the index, and competes with live traffic for exactly the rows being contended. A partition drop is a metadata operation.

Prefer a natural key where one exists

The separate dedupe table is the general mechanism, not always the best one. If the operation already carries a unique business identity, put the constraint on the real table and skip the indirection entirely:

create unique index on shipments (order_id);          -- one shipment per order
create unique index on ledger_entries (transfer_id);  -- one entry per transfer

Now the insert of the effect is the dedupe. There is no second table to size, expire, or keep consistent, and the invariant is enforced for every code path — including the batch job someone writes next quarter that forgets your API layer exists. A client-supplied idempotency key is the fallback for operations with no natural identity, not the default.

The trade-off is the error surface: a unique-violation on shipments tells you a shipment exists but not what the original response was, so you have to go read it. That is usually fine, and worth the simplicity.

When the effect leaves your database

Everything above relies on the effect and the dedupe record committing atomically. The moment the effect is an HTTP call to a payment processor, an email send, or a message publish, that is gone and you need a different shape.

The sequence is: commit an in_flight record naming the intent, make the external call with the same idempotency key forwarded downstream, then commit the result. If the process dies in the middle, a sweeper finds in_flight rows older than some threshold and replays them — and because the key was forwarded, the replay is safe. Every serious payments API accepts a client-supplied key for exactly this reason; passing yours through rather than minting a new one per attempt is the difference between a two-line recovery path and a reconciliation job.

For publishing to a queue, the same problem has a cleaner answer: write the event to an outbox table in the business transaction and let a relay publish it. The publish becomes at-least-once with an idempotent consumer, which is the standard arrangement described in message queues explained.

What idempotency does not solve

Ordering. Two different operations, each idempotent, can still arrive out of order. "Set status to shipped" then "set status to cancelled" produces a different end state than the reverse, and no key prevents it. Ordering needs sequence numbers or a state machine that rejects invalid transitions.

Concurrent conflicting writes. Idempotency answers "is this the same request again?" It says nothing about two different requests racing to update the same row — the lost-update problem. That needs optimistic concurrency: a version column or an If-Match ETag, so the second writer is told its view was stale.

Two intents that should have been one. A user who submits, reloads, and submits again generates two keys. The server is right to apply both. If "one order per cart" is the real invariant, it needs to be a constraint on the order table, not a property of the retry mechanism.

Staleness on replay. A replayed response is a snapshot of what the first attempt returned. If the charge has since been refunded, the replay still says 201 with the original body. That is correct behaviour and usually what you want, but it means the stored response is a receipt, not a read of current state.

Reconciliation. Deduplication catches the common case well. It does not catch a processor that recorded a charge you never learned about, or a bug that wrote the wrong amount. Money systems still need a double-entry ledger and a daily reconciliation against the processor's own records. Idempotency reduces the size of the exception report; it does not eliminate it.

The alternatives worth naming

Compare-and-set transitions. For state machines, skip keys entirely: update orders set status = 'shipped' where id = $1 and status = 'packed'. The affected-row count tells you whether you were the one who made the change, and a retry naturally affects zero rows. This is idempotency with no storage cost, and it is the right answer far more often than a key table. Reach for keys when you are creating something or moving money.

Absolute-state PUT instead of delta POST. If the client can name the resource and send its full desired state, repetition is free by construction. Designing the API this way is strictly better than bolting on a dedupe layer, and it is the first thing to ask whether you can do.

Conditional requests. If-Match with an ETag, or If-Unmodified-Since, gives you safe retries and conflict detection in one mechanism. It solves a different problem from idempotency keys — "don't apply this if something changed" rather than "don't apply this twice" — and real APIs often need both.

Server-minted request tokens. The old synchronizer-token pattern: the server issues a single-use token when it renders the form or starts the flow, and rejects a submission whose token is already spent. It is the one approach that handles the reload-and-resubmit case, because the token is tied to the server's notion of the attempt rather than the client's.

Transactional processing inside one system. Kafka's transactions give exactly-once processing for a closed read-process-write loop over Kafka topics, with the consumer offset committed alongside the output. It is genuine and it is also narrow: the moment a side effect leaves that boundary you are back to keys.

Saying this in an interview

Idempotency rarely gets a question of its own. It shows up the moment you say "the client retries" — on a payment flow, a chat send, an offline sync, a queue consumer — and the interviewer wants to know whether you noticed what you just committed to. The order that lands:

1. Name the ambiguity, not the solution. "A timeout doesn't tell the client whether the write landed, so the client has to retry, so the server has to make retries free."
2. Say where the key comes from. Client-generated, minted once per intent, scoped to the account.
3. Describe the mechanism in one sentence. "Insert the key and the effect in one transaction; a unique violation means a duplicate, and we replay the stored response."
4. Volunteer the race. Say out loud that check-then-act is wrong under concurrent retries and that the unique constraint is doing the arbitration.
5. State the retention. "Twenty-four hours, expired by dropping daily partitions" — a specific number signals you have run one of these.
6. Pre-empt the hard follow-up by naming what happens when the effect is external: an in_flight record, the key forwarded downstream, and a sweeper.

Step 6 is where the real signal is. Anyone can describe a dedupe table; the candidates who stand out are the ones who notice that the whole scheme rests on atomicity and immediately say what they would do when they cannot have it.

If you want to find out how that holds up when someone interrupts to ask what happens if the sweeper runs twice, you can practise it out loud in a voice system design mock on Whitepad.