A webhook endpoint that passes local tests falls apart during its first production week. Stripe, Shopify, and GitHub deliver events with an at-least-once guarantee: duplicates are certain, ordering is not promised, and a downed endpoint becomes a silent retry queue. The checklist below covers idempotency, retries, signatures, and monitoring so your stack receives every event without silent losses.

Why do webhooks arrive duplicated?

The at-least-once guarantee exists because senders cannot distinguish a network failure from a lost response. Your server processed the event and the 200 response vanished on the way back? The provider sends it again. Stripe keeps retrying with exponential backoff for up to 3 days; other providers run between 3 and 20 configured attempts. Duplicates are the normal case, so your handler has to be idempotent by construction.

Idempotency with a unique index

Store the provider-supplied event ID (Stripe's event.id) in a table backed by a unique index. One pattern covers the safe flow: open a transaction, insert the ID, bail out when it already exists, process the event, commit. Processing and recording inside the same transaction closes the gap where a crash between "processed" and "recorded" forces rework.

The event-ordering trap

Under retry pressure, a charge.refunded can land before the charge.succeeded that produced it. Treat every state transition as suspect until verified: a refund arriving for a payment you have never seen calls for a query against the provider API, which holds current truth. The webhook tells you it is time to look; the API answers.

HMAC signature validation

A public endpoint accepts payloads from anyone. Verify the HMAC signature providers attach (the Stripe-Signature header pattern) with timing-safe comparison: hash_equals() in PHP, Crypto.timingSafeEqual in Node.js. Reject events whose timestamps fall outside a 5-minute tolerance window, the typical cutoff that blocks replays of captured payloads.

Respond fast, process async

A handler that runs the entire business rule before returning 200 blows past the provider timeout, which triggers a resend, which triggers another long execution. The duplicate storm feeds itself. Acknowledge with 200 within seconds and push the event into a queue (SQS, Redis Streams, BullMQ); workers handle the heavy lifting under the same idempotency guarantees.

Monitoring and replay

  • Delivery success rate per endpoint, alerting when it drops below 99%
  • An alert whenever the dead-letter queue starts accumulating events
  • Manual replay tooling over the event log settles support tickets and debugging sessions

Sending webhooks of your own

On the sending side, sign every payload, document your retry policy, expose an event log with delivery status, and support secret rotation without downtime. Consumers depending on your webhooks need all four; without them, every incident turns into a support ticket.