Replay a failed event without repeating the business action

Received is not processed

Assume a payment provider sends events that create a downstream finance task and update a CRM record. A synchronous handler performs every action before returning. If one step fails or the response is lost, the provider may retry while the original action already exists.

Separate receipt from processing. Verify the provider's signature against the required request representation, validate the event and persist it durably before acknowledging successful receipt. A queue acknowledgment without durable storage can lose the event.

Stripe explicitly documents duplicate delivery and no guaranteed event order. Those are useful concrete constraints for this worked example; verify each other provider's own contract.

Two kinds of identity protect two different things

Use provider account plus event ID to deduplicate delivery. Claim that key atomically, for example with a database uniqueness constraint, rather than checking and inserting in separate steps that can race.

Also identify the business effect. Different events can request the same destination action. A task might use invoice ID plus action type; an invoice draft might use an approved milestone reference. Persist that effect key and the destination ID.

An event receipt marked complete too early can hide a failed effect. Track pending, processing, completed and review-needed states with attempt history. Only mark the effect complete after its destination result is known.

Check the full n8n production-readiness guide ↗

The hardest failure is an unknown outcome

A timeout after a create request doesn't tell you whether the destination created the record. First look up the external reference or use the destination's supported idempotency mechanism. If neither gives a reliable answer, route to operator review.

Retry transient failures with bounded backoff and a rate limit appropriate to the destination. Validation failures need corrected data. Expired credentials need an operator. Treating all errors as retryable turns a recoverable fault into an endless queue.

Events that arrive out of order should not blindly replace current state with an older snapshot. Retrieve current source state where appropriate or apply a version-aware transition rule.

Replay is an operation with a scope

After fixing the cause, select a bounded set of failed events. Record who authorized replay, why, the selection criteria and the expected destination effects. Run through the same identity checks as normal delivery.

Start with a small sample and inspect the results. Limit concurrency so recovery doesn't overload the system that just recovered. Repeated failures return to a review queue with attempt history rather than disappearing into another automatic loop.

Retain only the event data needed for diagnosis and replay under the organization's retention policy. Logs should carry identifiers and error categories without dumping credentials or unnecessary personal data.

Rehearse recovery before launch

The acceptance test is more than a successful event. Deliver the same event twice, send two distinct events for the same effect, lose the destination response and process a stale event after a newer one.

Operators need to know how to inspect the queue, pause a worker and verify destination state. If the integration platform already provides these controls, use them and document their limits.