Two kinds of failure, two responses
A transient failure is the network dropping, a provider being briefly overloaded, or a timeout. The same request, sent again in a minute, will probably work. Retry is the correct response.
A permanent failure is a rejection: an invalid phone number, a missing required field, a refused authorisation, a record that no longer exists. Retrying it produces the identical rejection at increasing cost. The correct response is to stop, mark the run failed, and put it in front of a human.
Backoff, and why the gaps grow
Retries should get further apart, not closer together. Immediate hammering of a struggling provider is how a small outage becomes your outage too. Exponential spacing — a minute, then two, then four — gives the other side room to recover.
There must also be a ceiling. After a fixed number of attempts the run settles permanently as failed. An automation that retries forever is a slow-motion incident nobody is looking at.
Idempotency in plain terms
If the system asks 'create this invoice' twice because the first response was lost, you want one invoice. An idempotency key is a stable identifier attached to the request: the receiving side recognises the repeat and returns the original result instead of doing the work again.
This is the only thing standing between a retry mechanism and duplicate records. Any action that creates something — a task, an invoice, a message — needs a key derived from the thing it represents, not a fresh random value per attempt.
The honest position on delivery guarantees
- At-most-once: might not happen. Safe for things you must never duplicate.
- At-least-once: will happen, possibly twice. Safe when paired with deduplication.
- Exactly-once: not achievable end-to-end across independent systems. Anyone claiming it is describing at-least-once plus deduplication.
- No rollback: a failed later step does not undo an earlier one. Order actions accordingly.