Tuesday afternoon load is easy. Saturday at 8 PM Eastern, when there are 12 NBA games, 14 NHL games, several major-league soccer matches, and a Champions League knockout tie all running concurrently, is when the architecture gets tested. This post walks through the patterns we use to keep a live sports prediction API responsive under peak game-night load, and the failure modes that only show up when everything spikes at once.
API request volume is not uniform. Three distinct patterns:
The third pattern is the one that breaks systems. Steady high load combined with bursty spikes exposes every shared bottleneck — thread pools, connection pools, cache eviction, GC pauses, log file I/O, downstream upstream rate limits.
An uncached prediction request runs the full inference path: fetch live game state, run model forward pass, compute calibration, build response. Several milliseconds of CPU per request. At 2000 req/sec that is potentially 8-10 cores fully saturated just on inference.
A cached request reads the precomputed prediction from memory and returns it. Sub-millisecond response time. Same 2000 req/sec drops to a single core.
The cache pattern we use is two-layered:
import time
from functools import lru_cache
# Layer 1: in-process cache, 5-second TTL
_cache = {}
TTL = 5.0
def predict_cached(sport: str, game_id: str):
key = (sport, game_id)
now = time.time()
if key in _cache and (now - _cache[key]["ts"]) < TTL:
return _cache[key]["data"]
data = predict_uncached(sport, game_id)
_cache[key] = {"data": data, "ts": now}
return dataFor multi-worker deployments, layer two is a Redis cache shared across workers. Same TTL, same key. The hot game on a Saturday night gets requested thousands of times per second; it computes once per 5 seconds and serves the rest from cache.
Global rate limits protect the service. Per-customer rate limits protect the service from any single customer accidentally (or intentionally) burning through its quota.
The pattern that has worked for us is a token bucket per customer, with bucket size proportional to their tier:
| Tier | Steady-state rate | Burst |
|---|---|---|
| Community | 1 req/sec | 5 |
| Starter | 10 req/sec | 30 |
| Pro | 100 req/sec | 250 |
| Enterprise | 1000 req/sec | 3000 |
Implementation is a Redis-backed leaky-bucket counter on the API gateway layer. Cheap, deterministic, and gives the customer a clear error message (with Retry-After header) when they exceed their tier.
The burst headroom matters: real customers do not poll evenly. A trading bot updates 50 watched games at the top of every minute and sits idle in between. Allowing a 5x burst handles the realistic pattern without forcing customers to add jitter.
For a cached prediction request, the budget is roughly:
| Step | Budget (p99) |
|---|---|
| TLS handshake (cached) | 5 ms |
| Auth + rate-limit check | 2 ms |
| Cache lookup | 1 ms |
| Response serialization | 1 ms |
| Network return | 10-50 ms (geo-dependent) |
| Total | 20-60 ms |
For a cache miss, add 5-20 ms for inference. Total p99 stays under 100 ms even on cold paths.
Latency above 100 ms is almost always one of three things: garbage collection (Python or otherwise), connection pool exhaustion, or downstream upstream stalls. Monitor each independently.
The most dangerous failure mode is silently slow. A request that takes 10 seconds blocks the worker; the worker queue backs up; eventually the API returns 503 to everything. By that point, customers have already retried multiple times, amplifying the load.
Our backpressure pattern enforces a hard request timeout at the worker level (3 seconds for cache miss, 1 second for cache hit). Any request that exceeds it gets cut and returns 503. Better to fail fast than hang slowly.
For graceful degradation under sustained overload, the API can shed load on lower-tier endpoints first. /v1/predictions/latest (full daily list) gets 503'd before /v1/predict/{sport}/{game_id} (single hot game). Trading bots get to keep working while research callers wait it out.
Every dependency you call has its own rate limit. ESPN's scoreboard endpoint, the odds API, the Polymarket WebSocket. If your background poller hits an upstream limit, your cache stops getting refreshed. If your cache stops getting refreshed, your API serves stale data — or, worse, returns 404 because the game is no longer in the cache.
The pattern we use is upstream-aware backoff. The poller maintains a per-upstream rate-limit budget. When 80% of the budget is consumed, it logs a warning. When 100% is hit, it backs off and falls over to a secondary source if available.
UPSTREAM_LIMITS = {
"espn": {"rate": 100, "window_sec": 60},
"the_odds_api": {"rate": 500, "window_sec": 86400}, # daily
"polymarket_clob": {"rate": 1000, "window_sec": 60},
}
class UpstreamGuard:
def __init__(self, name):
self.name = name
self.calls = []
def consume(self):
now = time.time()
cfg = UPSTREAM_LIMITS[self.name]
self.calls = [t for t in self.calls if now - t < cfg["window_sec"]]
if len(self.calls) >= cfg["rate"]:
raise UpstreamRateLimitExceeded(self.name)
if len(self.calls) >= 0.8 * cfg["rate"]:
logger.warning(f"{self.name} at 80% capacity")
self.calls.append(now)Failure modes that only show up at peak load:
Three operational pieces matter as much as the code:
Reliability under peak load is not an emergent property. It is the sum of intentional choices: cache aggressively, rate-limit per customer, enforce hard timeouts, monitor every upstream dependency, and degrade gracefully rather than failing all at once. Pick one practice from this post that you have not implemented and ship it before the next major game night.
ZenHodl serves calibrated probabilities for 11 sports under exactly the architecture described above. Free seven-day trial.
Try ZenHodl free