feat: Add EnvoyProxyTLSHandshakesFailing alert - #2098
Draft
ubergesundheit wants to merge 1 commit into
Draft
Conversation
Two incidents (2026-07-17 and 2026-07-28) saw Envoy proxies stop serving correctly while every deployed alert stayed silent: the Deployment reported 2/2 Ready and the Gateway stayed Programmed: True throughout. Nothing currently deployed observes the actual serving path, and the existing EnvoyProxySDSInitFetchTimeout only catches one specific cause (an SDS cert that never arrived). This alert is cause-agnostic: it catches any "listener up, serving broken" state -- expired cert, mis-issued cert, SDS starvation, anything -- by looking for TLS connection errors while zero handshakes succeed. The error rate is floored at 0.05/s rather than the obvious `> 0`. On an idle listener a single stray probe or aborted connection keeps `rate(error) > 0` true for a whole rate window while `rate(handshake) == 0` is legitimately true, so the unfloored form would have paged ~10 times over 7 days across gazelle and ferret, twice on production workload clusters. Measured over that window the error rate seen while handshakes were zero peaked at 0.0093/s, so the floor keeps a ~5x margin while staying far below the 0.1-1.1 handshakes/s of any listener that actually serves traffic. The two series carry an identical label set, so the implicit full-label match of `and` joins them exactly; an explicit `and on(pod, namespace, ...)` would be wrong here as it would drop envoy_listener_address and let an error on one listener pair up with an idle sibling listener on the same pod. Unit tests cover the healthy gazelle baseline, the idle-listener false-positive case that drove the floor, and the incident itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two incidents (2026-07-17 and 2026-07-28) saw Envoy proxies stop serving correctly while every deployed alert stayed silent: the Deployment reported 2/2 Ready and the Gateway stayed Programmed: True throughout. Nothing currently deployed observes the actual serving path, and the existing EnvoyProxySDSInitFetchTimeout only catches one specific cause (an SDS cert that never arrived).
This alert is cause-agnostic: it catches any "listener up, serving broken" state -- expired cert, mis-issued cert, SDS starvation, anything -- by looking for TLS connection errors while zero handshakes succeed.
The error rate is floored at 0.05/s rather than the obvious
> 0. On an idle listener a single stray probe or aborted connection keepsrate(error) > 0true for a whole rate window whilerate(handshake) == 0is legitimately true, so the unfloored form would have paged ~10 times over 7 days across gazelle and ferret, twice on production workload clusters. Measured over that window the error rate seen while handshakes were zero peaked at 0.0093/s, so the floor keeps a ~5x margin while staying far below the 0.1-1.1 handshakes/s of any listener that actually serves traffic.The two series carry an identical label set, so the implicit full-label match of
andjoins them exactly; an explicitand on(pod, namespace, ...)would be wrong here as it would drop envoy_listener_address and let an error on one listener pair up with an idle sibling listener on the same pod.Unit tests cover the healthy gazelle baseline, the idle-listener false-positive case that drove the floor, and the incident itself.
Before adding a new alerting rule into this repository you should consider creating an SLO rules instead.
SLO helps you both increase the quality of your monitoring and reduce the alert noise.
Towards: https://github.com/giantswarm/...
This PR ...
Checklist
oncall-kaas-cloudGitHub group).