Skip to content

feat: Add EnvoyProxyTLSHandshakesFailing alert - #2098

Draft
ubergesundheit wants to merge 1 commit into
mainfrom
add-envoy-proxy-tls-handshakes-failing-alert
Draft

feat: Add EnvoyProxyTLSHandshakesFailing alert#2098
ubergesundheit wants to merge 1 commit into
mainfrom
add-envoy-proxy-tls-handshakes-failing-alert

Conversation

@ubergesundheit

Copy link
Copy Markdown
Member

Two incidents (2026-07-17 and 2026-07-28) saw Envoy proxies stop serving correctly while every deployed alert stayed silent: the Deployment reported 2/2 Ready and the Gateway stayed Programmed: True throughout. Nothing currently deployed observes the actual serving path, and the existing EnvoyProxySDSInitFetchTimeout only catches one specific cause (an SDS cert that never arrived).

This alert is cause-agnostic: it catches any "listener up, serving broken" state -- expired cert, mis-issued cert, SDS starvation, anything -- by looking for TLS connection errors while zero handshakes succeed.

The error rate is floored at 0.05/s rather than the obvious > 0. On an idle listener a single stray probe or aborted connection keeps rate(error) > 0 true for a whole rate window while rate(handshake) == 0 is legitimately true, so the unfloored form would have paged ~10 times over 7 days across gazelle and ferret, twice on production workload clusters. Measured over that window the error rate seen while handshakes were zero peaked at 0.0093/s, so the floor keeps a ~5x margin while staying far below the 0.1-1.1 handshakes/s of any listener that actually serves traffic.

The two series carry an identical label set, so the implicit full-label match of and joins them exactly; an explicit
and on(pod, namespace, ...) would be wrong here as it would drop envoy_listener_address and let an error on one listener pair up with an idle sibling listener on the same pod.

Unit tests cover the healthy gazelle baseline, the idle-listener false-positive case that drove the floor, and the incident itself.

Before adding a new alerting rule into this repository you should consider creating an SLO rules instead.
SLO helps you both increase the quality of your monitoring and reduce the alert noise.


Towards: https://github.com/giantswarm/...

This PR ...

Checklist

Two incidents (2026-07-17 and 2026-07-28) saw Envoy proxies stop serving
correctly while every deployed alert stayed silent: the Deployment reported
2/2 Ready and the Gateway stayed Programmed: True throughout. Nothing
currently deployed observes the actual serving path, and the existing
EnvoyProxySDSInitFetchTimeout only catches one specific cause (an SDS cert
that never arrived).

This alert is cause-agnostic: it catches any "listener up, serving broken"
state -- expired cert, mis-issued cert, SDS starvation, anything -- by
looking for TLS connection errors while zero handshakes succeed.

The error rate is floored at 0.05/s rather than the obvious `> 0`. On an
idle listener a single stray probe or aborted connection keeps
`rate(error) > 0` true for a whole rate window while
`rate(handshake) == 0` is legitimately true, so the unfloored form would
have paged ~10 times over 7 days across gazelle and ferret, twice on
production workload clusters. Measured over that window the error rate
seen while handshakes were zero peaked at 0.0093/s, so the floor keeps a
~5x margin while staying far below the 0.1-1.1 handshakes/s of any
listener that actually serves traffic.

The two series carry an identical label set, so the implicit full-label
match of `and` joins them exactly; an explicit
`and on(pod, namespace, ...)` would be wrong here as it would drop
envoy_listener_address and let an error on one listener pair up with an
idle sibling listener on the same pod.

Unit tests cover the healthy gazelle baseline, the idle-listener
false-positive case that drove the floor, and the incident itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant