← Back to all engineering articles

Webhook Retries and Backoff: How Stripe, GitHub, and Shopify Handle Down Endpoints

When your backend goes down for maintenance or encounters a transient database lock during webhook processing, webhook providers don't just give up. They retry. And if your service isn't built to handle retries properly, you risk double-charging customers, provisioning duplicate licenses, or getting taken down by retry storms.

Understanding the retry schedules of major webhook providers — and designing your consumer endpoints to be idempotent — is the difference between an unnoticeable network blip and a severe production incident.

How Major Webhook Providers Retry

Each platform uses a slightly different backoff strategy:

The Anatomy of a Retry Storm

Imagine your webhook endpoint experiences a 10-minute database slowdown. During those 10 minutes, 1,000 webhook events fail with HTTP 504 Gateway Timeout:

The Cascade Effect:

  1. 1,000 webhook events fail at 12:00.
  2. Without jitter, all 1,000 providers retry simultaneously at 12:05.
  3. Your already-recovering database gets hit with 1,000 retries plus incoming new events.
  4. The database crashes again, pushing retries into an even larger synchronized wave.

Designing Idempotent Webhook Consumers

Because retries can occur even when your server actually processed the request (e.g., if the HTTP 200 response got dropped on the wire before reaching the provider), your handlers must be idempotent:

1. Store the Provider's Unique Event ID

Every major provider assigns a unique event ID to each event payload:

Before processing an event, check your database if this event ID has already been recorded:

// In your webhook consumer handler
func HandleWebhook(w http.ResponseWriter, r *http.Request) {
    eventID := r.Header.Get("X-GitHub-Delivery")
    
    // 1. Check if already processed
    processed, err := db.HasProcessedEvent(r.Context(), eventID)
    if err != nil {
        http.Error(w, "internal error", http.StatusInternalServerError)
        return
    }
    if processed {
        // Acknowledge immediately with 200 OK without re-running actions
        w.WriteHeader(http.StatusOK)
        return
    }

    // 2. Process the event inside an atomic transaction
    if err := db.ProcessOrderAndRecordEvent(r.Context(), eventID, payload); err != nil {
        http.Error(w, "processing failed", http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusOK)
}

2. Acknowledge Fast (Under 5 Seconds)

Most webhook providers timeout after 5 to 15 seconds. If your webhook handler initiates long-running tasks (like PDF generation or sending transactional emails), do not block the HTTP response.

Validate the signature, persist the event to an internal job queue (or Redis/Postgres queue), and reply 200 OK immediately.

Testing Retry Behavior Safely

The biggest blind spot in webhook development is assuming that retries only happen when your code crashes.

With SnapHook, you can test how your upstream provider behaves when your endpoint is unavailable:

# Configure a synthetic 500 error on your capture session
curl -X PUT https://snaphook.site/api/{slug}/response \
  -H 'Content-Type: application/json' \
  -d '{"status":500,"body":"simulated upstream outage"}'

Trigger an event in Stripe or GitHub, and observe the retry cadence in the live inspector:

Rule of Thumb:

Never deploy a webhook integration to production without intentionally testing a failure state first. A webhook integration that works on the happy path is only half tested.


Inspect your webhooks live in 2 seconds

No signup required. SnapHook creates a dedicated URL, streams incoming HTTP payloads over SSE, and lets you customize synthetic status codes and latency.

Next Up from the Blog

Engineering 7 min read

Stop Guessing What Your Webhooks Actually Send

Why ngrok tunnels, local loopbacks, and temporary console.log server deploys break down when inspecting production webhooks, and how in-memory SSE ring buffers provide zero-leak, zero-install ephemeral capture.