In the complex ecosystem of containerized applications, resource availability is a double-edged sword. While Kubernetes is designed to scale elastically, it also enforces strict limits via CPU and memory quotas defined in Pod specifications. When these limits are hit, the system reacts in two distinct ways: throttling, which degrades performance silently, and Out-of-Memory (OOM) kills, which terminate pods abruptly. Detecting these issues early is critical for maintaining Service Level Objectives (SLOs).
Many organizations rely on default monitoring dashboards, but they often lack the specificity required for production-grade alerting. Default rules might tell you a pod is down, but they rarely explain why or warn you about the silent performance degradation caused by CPU throttling before it causes an outage. This guide details how to craft custom Prometheus alerting rules to specifically target resource contention and memory exhaustion events.
Understanding the Metrics for Throttling
To alert on CPU throttling, we must look at the container runtime metrics exposed by kubelet. The most common metric used is container_cpu_cfs_throttled_seconds_total from the container_metrics API. This counter increments only when a container is throttled because it has exceeded its CPU quota.
A simple threshold is often insufficient. A spike in throttling might be transient during a startup phase, so we need to look at the rate of change over a specific window. We calculate the rate of throttled seconds to determine how much CPU time is being lost per second. If the rate exceeds a defined threshold (e.g., 0.1 seconds lost per second over a 5-minute window), it indicates a sustained resource shortage that will likely cause latency spikes.
Detecting Silent OOM Risks
While an OOM kill is obvious (the pod restarts), detecting the risk of an OOM kill before it happens is the gold standard of SRE practices. Kubernetes exposes memory usage metrics such as container_memory_working_set_bytes. However, we don't just want to alert when memory is high; we want to alert when memory is approaching the limit.
The strategy involves comparing the current memory usage against the defined limit. If a pod's memory usage consistently stays above 85% of its limit for more than 10 minutes, it is highly probable that the application will soon trigger an OOM kill. This "pre-mortem" alert allows operators to scale the resource request, increase the limit, or investigate a memory leak before the application crashes.
Constructing the Prometheus Alert Rules
Let's put this into practice by defining a prometheus_alerts.yaml configuration. This rule set targets both CPU throttling and memory pressure scenarios.
groups:
- name: k8s-resource-alerts
interval: 30s
rules:
# Alert on CPU Throttling
- alert: HighCPUThrottling
expr: |
rate(container_cpu_cfs_throttled_seconds_total{namespace="prod"}[5m])
/
rate(container_cpu_cfs_period_seconds_total{namespace="prod"}[5m])
> 0.05
for: 5m
labels:
severity: warning
team: platform
annotations:
summary: "High CPU throttling detected in {{ $labels.pod }}"
description: "Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} is experiencing significant CPU throttling."
# Alert on Memory Pressure (Risk of OOM)
- alert: HighMemoryUsage
expr: |
(container_memory_working_set_bytes{namespace="prod"}
/
container_spec_memory_limit_bytes{namespace="prod"}) * 100 > 85
for: 10m
labels:
severity: critical
team: platform
annotations:
summary: "Memory pressure detected in {{ $labels.pod }}"
description: "Pod {{ $labels.pod }} is using {{ $value }}% of its memory limit."Refining for Accuracy
Notice the use of the rate() function in the CPU rule. We divide the throttled seconds by the period seconds to get a ratio, which effectively normalizes the data regardless of the node's CPU architecture. In the memory rule, we handle the case where limits might be infinite by ensuring the denominator is valid, though Prometheus handles division by zero gracefully in many versions by returning NaN, which we can filter out with additional logic if needed.
Additionally, always scope your rules to specific namespaces or labels (like prod) to prevent alert fatigue from dev environments. Use for clauses to ensure that alerts are only fired if the condition persists, filtering out transient spikes that are often harmless.
Conclusion
Building custom alerting rules transforms your monitoring stack from a passive observer into an active guardian of your infrastructure. By specifically targeting CPU throttling ratios and memory pressure percentages, you shift from reacting to outages to preventing them. These custom rules provide the granular visibility needed to maintain high availability in Kubernetes environments, ensuring that your applications remain performant and stable under load. Invest time in tuning these metrics, and your SRE team will thank you for the reduced noise and faster incident response times.