Webhook retries and idempotency

Why duplicates are unavoidable

Your handler finishes, writes to the database, and returns 200. The response is lost in transit.

The provider now knows only that it did not get an answer. It cannot distinguish “the request never arrived” from “the request was processed and the reply was lost”, and those two cases need opposite responses. Retrying risks doing the work twice; not retrying risks never doing it at all.

Every provider chooses to retry. That choice is what makes idempotency your problem rather than theirs, and no amount of care on your side changes it — the ambiguity is in the network, not in the code.

The four sources of duplicates

A lost or slow response. The case above. Also fires when your handler exceeds the provider’s deadline: the work completed, the acknowledgement arrived too late, and the provider retries an event you have already handled.

Multiple subscriptions. Two endpoints in the dashboard covering the same event type, or a wildcard subscription alongside a specific one, and each fires independently. Common after a migration where the old endpoint was never deleted.

Several events for one state change. A Stripe checkout produces checkout.session.completed, payment_intent.succeeded, charge.succeeded, and invoice.paid. These are four distinct events with four distinct ids — a per-id check will not deduplicate them, because they are not duplicates. They are four notifications about one payment, and only one of them should trigger fulfilment.

Provider-side redelivery. Someone pressed resend in the dashboard, or the provider replayed a backlog after an outage. Same payload, sometimes the same id, sometimes a new one.

The first, second, and fourth are handled by an id check. The third is not, and it is the one that ships bugs.

What each provider actually does

Provider	Retries	Window	On sustained failure
Stripe	Exponential backoff	~3 days	Emails you; endpoint may be disabled
GitHub	None	—	Delivery is simply lost
Shopify	8 attempts	~48 hours	Endpoint removed after 19 hours of failure
Slack	3 attempts	Minutes	Event subscription disabled
Razorpay	Several, with backoff	Up to 24 hours	Stays enabled
Paddle	3 attempts	~1 hour	Notification marked failed
Clerk / Svix	5 attempts	~24 hours	Endpoint disabled after 5 days
Twilio	Configurable	Short	Depends on the setting
Mailgun	8 attempts	~8 hours	Webhook disabled
Supabase auth hooks	None	—	The user’s sign-in fails

Two rows deserve attention. GitHub does not retry — if your endpoint is down for a deploy, those pushes are gone, and the only recovery is the Redeliver button in Recent Deliveries. Supabase auth hooks run inside the user’s request, so a failure is not a queued retry; it is a person unable to sign in.

Everywhere else, the practical rule is: assume a bounded window, assume no ordering, and assume you may be retried after you have already succeeded.

The idempotency pattern

Use the provider’s event id and a unique constraint. Not a SELECT followed by an INSERT — two concurrent retries both read “not present” and both proceed.

CREATE TABLE processed_events (
  event_id   TEXT PRIMARY KEY,
  event_type TEXT NOT NULL,
  handled_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

async function handle(event) {
  try {
    await db.query(
      'INSERT INTO processed_events (event_id, event_type) VALUES ($1, $2)',
      [event.id, event.type],
    );
  } catch (err) {
    if (err.code === '23505') return;  // unique_violation — already handled
    throw err;
  }

  await doTheWork(event);
}

The order matters and it is a genuine trade-off. Recording first means a crash between the insert and the work leaves the event marked handled but not done. Recording after means a crash leaves it unmarked, and the retry does the work twice.

Neither is free, so pick deliberately:

• Record first when a duplicate would be worse than a miss — charging a card, sending an email, provisioning a licence.
• Record after, in the same transaction as the work, when the work is a database write. Then the two commit or roll back together and the trade-off disappears.

await db.transaction(async (tx) => {
  await tx.insert('processed_events', { event_id: event.id, event_type: event.type });
  await tx.update('orders', { status: 'paid' }, { id: orderId });
});

This is the version to reach for whenever it applies, because it is the only one with no window at all.

Beyond the id: making side effects safe

An id check protects against the same event twice. It does not protect against two different events describing one thing, and that is where the money goes.

Outbound API calls. Send your own idempotency key, derived from the event id:

await stripe.refunds.create(
  { charge: chargeId },
  { idempotencyKey: `refund-${event.id}` },
);

Stripe, Adyen, and most payment APIs support this. The second call returns the first call’s result instead of creating a second refund.

Database writes. Prefer an upsert over an insert, and prefer a state assignment over an increment:

-- safe under any number of duplicates
UPDATE orders SET status = 'paid', paid_at = $2 WHERE id = $1 AND status <> 'paid';

-- not safe: runs twice, counts twice
UPDATE accounts SET credits = credits + 100 WHERE id = $1;

If you must increment, make the increment itself conditional on a row in processed_events.

Emails and notifications. Check current state rather than reacting to the event. “Send the receipt if no receipt has been sent for this order” survives duplicates; “send the receipt because a payment event arrived” does not.

Fulfilment. Pick exactly one event as the trigger and ignore the rest. For Stripe Checkout that is checkout.session.completed with payment_status === 'paid'; the accompanying payment_intent.succeeded and charge.succeeded are informational. Write down which event is authoritative for each flow, because the next person to touch the handler will otherwise add a second trigger.

Ordering

Nothing guarantees order. A retry of event A can arrive after event B, and providers with parallel workers reorder without any failure involved.

So do not trust arrival order. Compare the timestamp inside the payload against what you have stored, and discard anything stale:

if (event.created * 1000 <= order.updatedAt.getTime()) return;  // stale, ignore

The failure this prevents is expensive and easy to reproduce: subscription.updated (cancelled) arrives, then a retry of an earlier subscription.updated (active) lands afterwards, and a handler that trusts arrival order has just reactivated a cancelled subscription.

For anything where correctness is not negotiable, treat the webhook as a signal rather than as data — fetch the current object from the API and write that. It costs a request and removes ordering from the problem entirely.

When the retries run out

Every window closes. Plan for the events that fall outside it.

Queue on arrival. Verify, push the raw body onto a durable queue, return 200. Now the provider’s retry window only has to cover your ability to accept a request, not your ability to process one — and your own retries are under your control, with your own dead-letter queue.

Reconcile periodically. A nightly job that lists objects changed since the last run and compares them against your database catches everything: dropped deliveries, events from before the endpoint existed, and the ones lost while you were deploying. For most integrations this is a few dozen lines and it is the only real safety net.

Watch for disabled endpoints. Shopify, Slack, Clerk, and Mailgun all disable a failing endpoint eventually, and Stripe emails you first. A quiet integration that used to be busy is a symptom, not good news — alert on the absence of expected events, not just on errors.