# Failure rehearsal matrix

Worked example companion. Fictional scenario; no measured results.

Before retrying, find out which destination records already exist.

## How to use this sheet

Replace the example rules with your own. Keep each acceptance decision linked to its owner and evidence. Use the blank CSV to record your actual process; the example CSV contains teaching material.

| Injected failure | Expected behavior | Evidence |
| --- | --- | --- |
| Same event delivered twice | One receipt claim and one effect | Event key and effect key |
| Distinct events request same action | Reuse existing business result | Destination reference |
| Response lost after create | Lookup before another create | Recovered destination ID |
| Older event arrives last | Do not regress current state | Source version or fetched state |
| Credentials expire | Pause or hold for repair | Owned error record |
| Replay still fails | Return to bounded review queue | Attempt history and reason |

## Workflow

1. **Verify and persist** (Receiver): Atomically claim provider account plus event ID.
2. **Process effect** (Worker): Use a stable business-action key.
3. **Confirm destination** (Destination adapter): Store result or investigate unknown outcome.
4. **Recover deliberately** (Operator): Repair cause and replay a bounded selection.

Exception: Unknown create outcomes require destination lookup. Permanent errors enter review. Replays reuse the original event and business-effect identities.

## Measurement

For a defined event cohort, report pending age, failed effects, duplicate effects and successful recoveries. Record the failure modes exercised. A local rehearsal doesn't establish a production availability percentage.

## Limitations

- Destination idempotency, lookup support and provider retention determine recovery options.
- No architecture can guarantee exactly-once effects across arbitrary APIs without supporting destination controls.

Source page: https://www.opsautomators.com/work/webhook-recovery-replay
Prepared September 7, 2026.