In the world of high-availability system design, the "Cache Stampede" (also known as the "Thundering Herd") is one of the most insidious performance killers. It occurs when a high-traffic cache key expires simultaneously, causing thousands of concurrent requests to bypass the cache and hit the backend database or upstream service. This sudden surge can overwhelm your infrastructure, leading to increased latency, database connection pool exhaustion, and potential service outages.
For intermediate to advanced developers, simply setting a short expiration time is not a viable solution due to the high hit-rate requirements of modern applications. Instead, we must implement robust strategies to mitigate this risk. In this post, we will explore two complementary techniques: Distributed Locking and Probabilistic Early Expiration.
The Anatomy of a Cache Stampede
To solve the problem, we must first understand its mechanism. Consider a scenario where a popular product page has a cache TTL (Time-To-Live) of 60 seconds. At t=60s, the key expires. If 10,000 users request this page at that exact millisecond, all 10,000 threads will check the cache, find nothing, and proceed to the database. Even if you have a database that can handle 1,000 queries per second, 10,000 concurrent queries will cause a backlog, potentially crashing the application.
The goal is to ensure that only one request is responsible for regenerating the data, while other requests wait or receive a stale version.
Strategy 1: Distributed Locking with Redlock
The most straightforward approach is to acquire a distributed lock before attempting to regenerate the cache. If the lock is acquired, the request generates the data and updates the cache. If the lock is already held, the request waits briefly or returns a cached stale value.
When implementing this in Redis, it is crucial to use atomic operations to avoid race conditions. The SET command with NX (Set if Not eXists) and EX (Expiration) flags is the standard way to implement a lock safely.
import redis
def get_data_with_lock(redis_client, key, data_fetcher):
# Attempt to acquire lock with NX and expiration to prevent deadlocks
# EX sets the TTL in seconds, ensuring the lock is released if the process crashes
if redis_client.set(lock_key, "locked", nx=True, ex=5):
try:
# Only one thread enters here
data = data_fetcher()
redis_client.set(key, data, ex=60) # Set cache TTL
return data
finally:
# Release the lock
redis_client.delete(lock_key)
else:
# Another thread is refreshing the cache
# Option A: Wait and retry (blocking)
# Option B: Return stale cache or error (non-blocking)
return redis_client.get(key)
While effective, this approach introduces latency for all waiting threads. In high-concurrency scenarios, blocking threads can consume significant resources. A more elegant solution for read-heavy workloads is Probabilistic Expiration.
Strategy 2: Probabilistic Early Expiration
Instead of letting the cache expire abruptly at t=60, we can use a probabilistic approach. We set the cache TTL to a variable range, such that expiration is likely to occur slightly before the nominal time. This staggers the regeneration requests across a time window, smoothing out the load on the database.
We can also implement a "lazy refresh" mechanism. When a key is accessed, we check if it is nearing expiration (e.g., within the last 10% of its TTL). If it is, we spawn a background thread or task to refresh the data asynchronously, while the current request returns the existing (slightly stale) data.
import time
import random
def get_data_probabilistic(redis_client, key, data_fetcher):
data = redis_client.get(key)
if not data:
# Cache miss: regenerate and set
new_data = data_fetcher()
# Add a small random jitter to TTL (e.g., 60s to 70s)
ttl = random.randint(60, 70)
redis_client.set(key, new_data, ex=ttl)
return new_data
# Cache hit: check if we should refresh early
ttl_remaining = redis_client.ttl(key)
# If remaining TTL is less than 10% of original max TTL, refresh asynchronously
if ttl_remaining < 7:
# In production, use a message queue or async task runner here
schedule_async_refresh(key, data_fetcher)
return data
def schedule_async_refresh(key, data_fetcher):
# This function should run in a separate thread/process
# to avoid blocking the main request thread
new_data = data_fetcher()
redis_client.set(key, new_data, ex=60)
Conclusion
Preventing cache stampedes is not about choosing one perfect solution, but about selecting the right tool for your specific traffic patterns. Distributed locking provides strong consistency but may introduce latency. Probabilistic expiration offers high availability and low latency at the cost of slight data staleness.
For many production systems, a hybrid approach is best: use short TTLs with jitter to spread out expiration, combined with asynchronous background refreshes. By implementing these strategies, you ensure that your infrastructure remains resilient under the heaviest loads, maintaining a smooth user experience even during traffic spikes.