Webhook Design That Will Not Wake You at 3 AM: Idempotency, Retries, and Signing

Webhooks are the most common way services talk to each other in production. They are also the most common cause of mysterious bugs at 3 AM. Duplicate orders, missing payment confirmations, replay attacks, and retry storms all come from webhooks designed with the happy path in mind.

This is a guide to making your webhooks boring. Boring is good. Boring means you sleep through the night.

The Three Failure Modes Every Webhook Hits

1. Duplicate deliveries. The sender sent the webhook, your server processed it, your ack timed out, the sender retries. You now have two records.
2. Out-of-order delivery. Webhook B (a refund) arrives before Webhook A (the original charge). Your code assumes order. It breaks.
3. Forged or replayed events. Someone intercepts a webhook and replays it 1,000 times, or crafts a fake one that looks legitimate.

Every pattern in this post addresses one of these three failure modes.

Pattern 1: Idempotency Keys

Stripe popularized this and it is now the industry standard. Every webhook payload carries a unique event ID (Stripe calls it event.id, others call it delivery_id). Your server stores a record of every ID you have processed. Before doing real work, you check:

const processed = await db.processed_webhooks
  .findOne({ event_id: payload.id })

if (processed) {
  return { status: 200, body: 'already processed' }
}

await processWebhook(payload)
await db.processed_webhooks.insertOne({
  event_id: payload.id,
  processed_at: new Date()
})

Two production-grade refinements:

• Use a unique constraint on event_id in the database. If two workers race, one will fail the insert and you can detect that as "already processed".
• Set a TTL on the table. Stripe events older than 30 days will not retry, so storing them forever wastes space. We typically use 60 to 90 days.

Pattern 2: Signature Verification

Every webhook should be signed by the sender with HMAC. Most providers use HMAC-SHA256. Verify the signature before doing anything else, including parsing the body.

Common mistake we see: parsing the JSON, then verifying the signature against the parsed object. The signature is over the raw body bytes. Re-serializing changes whitespace, key order, or encoding. The signature will fail. Verify against the raw body string.

app.post('/webhook', express.raw({ type: 'application/json' }), (req, res) => {
  const sig = req.headers['x-signature']
  const body = req.body  // raw Buffer
  const expected = crypto
    .createHmac('sha256', process.env.WEBHOOK_SECRET)
    .update(body)
    .digest('hex')

  if (!crypto.timingSafeEqual(Buffer.from(sig), Buffer.from(expected))) {
    return res.status(401).send('invalid signature')
  }

  const payload = JSON.parse(body)
  // proceed
})

Use timingSafeEqual, not ===. Timing attacks on signature comparison are a thing.

Pattern 3: Reject Old Events

Sign the webhook with a timestamp included in the payload (or a header) and reject anything more than 5 minutes old. This prevents replay attacks where someone captures a valid webhook and replays it later.

Stripe wraps this into the signature itself: t=1234567890, v1=abcdef.... The signature includes the timestamp, so changing one breaks the other.

Pattern 4: Acknowledge Fast, Process Async

Senders retry on slow responses. If your webhook handler does heavy work synchronously (sending an email, generating a PDF, hitting another API), the sender may time out and retry, even though you eventually succeeded. You end up doing the work twice.

The pattern: validate the signature, store the event in a queue, return 200 immediately. Then a background worker processes the queue. We use BullMQ on Redis for this; SQS or Cloud Tasks works equally well.

Pattern 5: Make Handlers Idempotent

Idempotency keys protect against duplicate webhook deliveries. But your handler logic should also be idempotent in case you receive an event twice through any other path (manual replay, debugging, etc.).

Concretely: write upserts, not inserts. Check before sending emails. Use external keys when creating downstream records. Every action your handler takes should be safe to repeat.

Pattern 6: Out-of-Order Tolerance

Webhooks rarely arrive in strict order. Pattern: each event includes a logical timestamp or sequence number. Before applying state changes, check whether you have already applied a newer event. If so, skip.

For account state events (subscription updated, user profile changed), we store last_event_at on the record and only apply changes from events newer than that.

What We Always Add to Production Webhook Setups

• A test endpoint that lets the sender prove they are reachable. Stripe has /webhook/test, GitHub has the ping event. Lets ops verify connectivity without firing real events.
• A dead-letter queue for events that fail after 5 retries. Better than losing them silently.
• Structured logs with the event ID, source, signature status, and processing duration. Pull this into Datadog or Grafana for alerting.
• A small admin page that lists recent webhook deliveries with replay buttons. Saves hours during incidents.

Closing Thoughts

The patterns above turn webhooks from a flaky integration concern into a boring infrastructure concern. We use this exact playbook on every webhook integration we ship. If you are wiring up Stripe, Paystack, GitHub, Slack, or any other webhook source and want it to be the kind of integration you forget exists, our API integration team can build it right the first time. Tell us what you are integrating and we will respond within 24 hours.