Alerting
Purpose of Alerting
Goal: Turn SLOs (Service Level Objectives) into actionable alerts.
Benefit: Respond to issues before consuming too much of the error budget.
Alerting Considerations
Precision: Detect significant events without false alerts.
Recall: Ensure all significant events trigger alerts.
Detection Time: Minimize the time taken to detect issues.
Reset Time: Ensure alerts resolve quickly once issues are fixed.
Six Approaches to Alerting
1. Target Error Rate ≥ SLO Threshold
Implementation: Trigger alerts if the error rate exceeds SLO for a short window (e.g., 10 minutes).
Pros: Simple, quick detection.
Cons: Poor precision—too many false positives for minor issues.
2. Increased Alert Window
Implementation: Increase the alert window (e.g., 36 hours) to improve precision.
Pros: Better precision than short windows.
Cons: Long reset times and higher memory costs.
3. Incrementing Alert Duration
Implementation: Only trigger alerts if the error rate stays above the threshold for a set duration.
Pros: Better precision for sustained issues.
Cons: Poor recall and slow detection time for severe issues.
4. Alert on Burn Rate
Definition: Burn rate = how fast the service consumes the error budget.
Implementation: Alert when the burn rate exceeds a critical threshold (e.g., burn rate of 36).
Pros: Good precision and detection time.
Cons: May miss lower, slow-moving errors.
5. Multiple Burn Rate Alerts
Implementation: Use multiple burn rates for different error severities (e.g., 2% budget in 1 hour, 5% in 6 hours).
Pros: Adaptive alerting for both fast and slow errors; prioritizes alerts based on severity.
Cons: More complex to manage multiple burn rates and windows.
6. Multiwindow, Multi-Burn-Rate Alerts (Recommended)
Implementation: Combine short and long windows with multiple burn rates (e.g., alert if both 1-hour and 5-minute windows exceed burn rate thresholds).
Pros: Best approach for managing precision, recall, detection, and reset times; reduces false positives.
Handling Low-Traffic Services
Problem: High sensitivity to errors in low-traffic services causes false alerts.
Solutions:
Generate artificial traffic to simulate user activity.
Combine smaller services into a single alerting system.
Modify product design to reduce impact of individual failed requests.
Handling Extreme Availability Goals
Low Availability: E.g., 90% availability—errors may go unnoticed in long error budgets.
High Availability: E.g., 99.999% availability—100% outages can deplete error budgets in seconds.
Solution: Design systems to avoid 100% outages or implement gradual rollouts (e.g., canarying).
Scalable Alerting Framework
Avoid Custom Parameters: Don't specify alert parameters for every microservice.
Group Requests into Buckets:
Critical: E.g., login requests (99.99% availability).
High Priority: E.g., user interaction (99.9% availability).
Low Priority: Non-urgent requests with minimal user impact.
Conclusion
Best Strategy: Multiwindow, multi-burn-rate alerting is the most reliable way to defend SLOs.
Objective: Set alerts to notify for actionable, significant events that impact the error budget.
Last updated