Overview
Network interruptions, deployments and busy origins can all cause failures. A useful recovery policy identifies the cause before deciding whether to wait, retry, change origin or return an error.
Available health checks, retry conditions and fallback configurations depend on the actual integration. These design principles do not imply that every failed request can be recovered automatically.
Classify the failure before retrying
Connection failures, read timeouts and application rejections mean different things. Authorization and validation errors should not normally be retried as though they were network failures.
| Situation | Evaluate | Possible response |
|---|---|---|
| Connection failure | Reachability and alternative sources | Try an available source within a bounded budget |
| Slow response | Application processing and total wait | Distinguish connection and read timeouts |
| Origin error | Retryable status and possible side effects | Decide according to request semantics |
| Permission or input error | Whether the application response is correct | Preserve the result instead of switching pointlessly |
Set a total time budget for a request
Several individually reasonable attempts can still make a user wait too long. Work backward from the intended experience when setting retries, delays and timeouts.
- Limit retries so an overloaded origin does not receive multiplied traffic.
- Review every proxy layer to avoid overlapping retry policies.
- Confirm idempotency and deduplication before retrying payments, transactions or writes.
Make sure a fallback can serve the application
A reachable address is only the beginning. Data, certificates, sessions and release versions must also be ready before it can take over requests.
Check application health
Use checks that represent critical service behavior without adding unnecessary load.
Verify consistency
Understand replication lag, session storage and content versions across origins.
Plan recovery
Require recovery confirmation and a gradual return procedure to avoid repeated switching near a failure threshold.
Implementation & checks
- 01
Establish a baseline
Record origin response patterns, errors and the application's acceptable waiting time.
- 02
Exercise realistic failures
Test disconnections, slow responses and an unavailable origin in a controlled environment, including duplicate-write checks.
- 03
Keep switching records
Connect failure reasons, actual attempts and final results so rules can be refined.