Scalability and Load Balancing
Scaling strategies
| Vertical (scale up) | Horizontal (scale out) | |
|---|---|---|
| How | Bigger machine | More machines |
| Limit | Hardware ceiling, single point of failure | Needs a distributed design |
| Complexity | Low | Higher |
Design services to be stateless so any instance can handle any request; keep state in a database, cache or session store.
Load balancers
A load balancer spreads requests across healthy servers and removes failed ones using health checks.
| Layer | Operates on | Notes |
|---|---|---|
| L4 | IP and port (TCP/UDP) | Fast, content-agnostic |
| L7 | HTTP (path, headers, cookies) | Routing, TLS termination, A/B routing |
Balancing algorithms
| Algorithm | Idea |
|---|---|
| Round robin | Rotate through servers |
| Weighted round robin | More traffic to bigger servers |
| Least connections | Send to the least busy server |
| IP / consistent hashing | Same client or key goes to the same server |
round_robin_lb.py
from itertools import cycle
servers = cycle(["10.0.0.1", "10.0.0.2", "10.0.0.3"])
def pick():
return next(servers)
Consistent hashing
Plain hash(key) % N remaps almost every key when N changes. Consistent hashing places servers and keys on a ring; a key belongs to the next server clockwise, so adding or removing a server only moves a small share of keys. Virtual nodes even out the distribution.
consistent_hash.py
import bisect, hashlib
class Ring:
def __init__(self, nodes, vnodes=100):
self.ring = sorted(
(self._h(f"{n}#{i}"), n) for n in nodes for i in range(vnodes)
)
self.keys = [h for h, _ in self.ring]
@staticmethod
def _h(s):
return int(hashlib.md5(s.encode()).hexdigest(), 16)
def node_for(self, key):
i = bisect.bisect(self.keys, self._h(key)) % len(self.ring)
return self.ring[i][1]
CDN
A content delivery network caches static content at edge locations near users, cutting latency and origin load. Use long Cache-Control lifetimes with versioned file names.
Loading Comments...