Webhooks arrive out of order

A cancellation that arrives before the renewal it precedes will, in a naive handler, cancel a subscription that is currently paid.

Wiring up billing means writing a webhook handler: the provider posts events to your endpoint, you grant access when payment succeeds, extend it when a subscription renews, revoke it when it ends. Written as a list, it reads like a state machine with obvious transitions.

Then you read the delivery guarantees and discover the list is not a sequence. It is a set of messages that will mostly arrive, mostly once, mostly in order.

Three things that are true of every provider

Retries mean duplicates

If your endpoint is slow, returns an error, or your host has a bad thirty seconds, the provider retries. Your handler will see the same event twice, and sometimes long after the first attempt. Any handler that does something incremental — adding days to an expiry, incrementing a counter, sending a receipt — will do it twice.

Order is not guaranteed

Two events generated a second apart can arrive in either order, because they may be delivered by different workers with different retry histories. The sequence you drew on paper is the sequence of things happening at the provider, not the sequence of things arriving at you.

Event names are not obvious

This one is prosaic and bites hardest. Documentation, SDKs, support emails and blog posts disagree about exact event names — succeeded against successful, updated against changed. A handler listening for an event the provider never sends does not error. It waits forever, and the feature it powers silently never happens.

The rule that fixes most of it

Treat the payload as the truth about current state, not the event name as a truth about transition.

Payment providers include the full current state of the object in each event — the subscription's status, its current period end, its plan. That state is authoritative as of when the event was generated. So rather than deriving state from which event arrived, read the state that came with it.

naive        on subscription.renewed  ->  expiry = expiry + 30 days
             (arrives twice: 60 days granted)

better       on any subscription event ->  expiry = payload.current_period_end
             (arrives twice: same value written twice)

The second version is idempotent by construction. Processing it twice does nothing the first did not. It also survives events arriving in the wrong order, because writing the same absolute value repeatedly converges regardless of sequence.

Ordering by timestamp, not arrival

Absolute values handle most cases; status changes need one more thing. If a cancellation and a renewal cross in flight, last-write-wins on arrival can leave you in the wrong state permanently.

Every event carries the time it was generated. Store that alongside the record, and ignore any event older than the one you last applied. Arrival order stops mattering; generation order is what you act on.

• StoreEvent id, to reject a duplicate you have already applied
• StoreGeneration timestamp, to reject anything stale
• ReadStatus and period end from the payload, never inferred from the event name
• Return200 quickly, then do the work — slow handlers cause the retries that cause duplicates

Fail towards the customer

A last principle, less technical. When your handler is uncertain — an event you do not recognise, a state you did not expect, a payload that does not parse — the safe direction is towards keeping access on, not switching it off.

A customer who keeps access for an extra day costs you a day. A paying customer locked out by a mis-sequenced webhook costs you the customer, and they will not write in to explain; they will assume your product is broken, which at that moment it is.

Grace periods exist for this. A failed payment should start a countdown, not a revocation. The provider is going to retry the charge anyway.