Reliability and Observability
Designing for failure
Everything fails: disks, networks, whole zones. Aim to avoid single points of failure and to degrade gracefully.
| Technique | Purpose |
|---|---|
| Redundancy / replication | Survive node or zone loss |
| Health checks and failover | Remove bad instances automatically |
| Timeouts | Never wait forever |
| Retries with backoff and jitter | Recover from transient errors without a retry storm |
| Circuit breaker | Stop calling a failing dependency and fail fast |
| Bulkheads | Isolate resources so one failure does not drain everything |
| Graceful degradation | Serve reduced functionality instead of an error |
retry.py
import random, time
def call_with_retry(fn, attempts=5, base=0.1):
for i in range(attempts):
try:
return fn()
except Exception:
if i == attempts - 1:
raise
time.sleep(base * 2 ** i + random.uniform(0, base))
CAUTION
Retries multiply load. Combine them with timeouts, a retry budget and circuit breakers, or a small outage can become a large one.
SLI, SLO and SLA
| Term | Meaning | Example |
|---|---|---|
| SLI | A measured indicator | 99.95% of requests succeed |
| SLO | The internal target | 99.9% monthly availability |
| SLA | A contractual promise with penalties | 99.5% or credits |
The error budget is 100% - SLO; it balances shipping speed against stability.
Observability
| Pillar | Answers | Tools |
|---|---|---|
| Metrics | How is it behaving? (rates, errors, latency) | Prometheus, Grafana |
| Logs | What happened? | ELK, Loki |
| Traces | Where did time go across services? | OpenTelemetry, Jaeger |
Track the golden signals: latency, traffic, errors and saturation. Alert on symptoms users feel, not on every internal cause.
Deployment safety
Blue-green and canary releases, feature flags, automated rollback, and database migrations that are backwards compatible.
Loading Comments...