feat(moq-relay-ietf): add metrics instrumentation by englishm-cloudflare · Pull Request #140 · cloudflare/moq-rs

englishm-cloudflare · 2026-02-04T18:14:50Z

Summary

Adds metrics instrumentation to moq-relay-ietf using the metrics crate facade. Metrics are always compiled in — when no recorder is installed, overhead is negligible (similar to the log crate pattern). An optional metrics-prometheus feature adds a Prometheus HTTP exporter for testing/standalone use.

Metrics

All metrics are prefixed with moq_relay_ to avoid collisions.

Counters (10)

Metric	Labels	Description
`moq_relay_connections_total`	-	Total incoming connections accepted
`moq_relay_connections_closed_total`	-	Total connections that have closed (graceful or error)
`moq_relay_connection_errors_total`	`stage`	Connection failures (session_accept, session_run)
`moq_relay_publishers_total`	-	Total ANNOUNCE requests
`moq_relay_announce_ok_total`	-	Successful ANNOUNCE_OK responses sent
`moq_relay_announce_errors_total`	`phase`	Announce failures (coordinator_register, local_register, send_ok)
`moq_relay_subscribers_total`	-	Total SUBSCRIBE requests
`moq_relay_subscribe_not_found_total`	-	Track not found after checking all sources
`moq_relay_subscribe_route_errors_total`	-	Infrastructure failure when routing to remote
`moq_relay_upstream_errors_total`	`stage`	Upstream connection failures (connect, session)

Gauges (6)

Metric	Description
`moq_relay_active_connections`	Current client connections
`moq_relay_active_publishers`	Current publishers
`moq_relay_active_subscriptions`	Current subscriptions
`moq_relay_active_tracks`	Tracks being served
`moq_relay_announced_namespaces`	Registered namespaces
`moq_relay_upstream_connections`	Upstream/origin connections

Histograms (1)

Metric	Labels	Description
`moq_relay_subscribe_latency_seconds`	`source`	Subscription resolution time (local, remote, not_found, route_error)

Implementation

Uses the metrics crate facade — consumers install their own recorder
GaugeGuard: RAII guard for automatic gauge increment/decrement
TimingGuard: RAII guard for automatic histogram duration recording
Both guards have #[must_use] to prevent accidental drops
All label values are &'static str — no cardinality explosion possible
Graceful close detection uses structural pattern matching on quinn::ConnectionError to correctly handle both raw QUIC and WebTransport close codes (HTTP/3 encoded)
Subscribe failures split into separate counters (not_found vs route_error) since they represent different failure modes (content problem vs infrastructure problem)
connections_closed_total properly incremented on all exit paths to maintain connections_total - connections_closed_total == active_connections invariant

Design Decisions

Always-on metrics (no feature flag): The metrics crate is zero-cost when no recorder is installed. This avoids 21+ #[cfg] annotations and matches how the log crate works. The metrics-prometheus feature remains for the optional Prometheus exporter.
Direct quinn dependency in moq-transport: Added for robust graceful close detection via structural matching on ConnectionError::ApplicationClosed. The previous string-matching approach failed for WebTransport connections where close code 0 is HTTP/3-encoded as 91343852333275. When transitioning to tokio-quiche, this should be updated.
Route errors return internal error: Instead of falling through to "not found", route errors now return an internal error to the subscriber — since we couldn't actually check if the track exists remotely.

Usage

The metrics crate is always available. To collect metrics, install a recorder before starting the relay:

// Example with prometheus exporter (requires metrics-prometheus feature)
metrics_exporter_prometheus::PrometheusBuilder::new()
    .with_http_listener("127.0.0.1:9090".parse().unwrap())
    .install()
    .expect("failed to install metrics exporter");

The standalone binary supports --metrics-addr when built with --features metrics-prometheus.

itzmanish · 2026-02-05T09:47:02Z

                }
                Err(e) => {
                    log::error!("failed to route to remote: {}", e);
+                    increment_counter!("moq_relay_subscribe_failures_total", "reason" => "route_error");


the second parameter is "reason" => "route_error" is that correct? should not that be "reason", "route_error"?

That's the syntax for adding labels, but your comment made me realize I'm not very happy with the increment_counter wrapper macro so I'm reworking things to use the upstream macros more directly for better discoverability. See: https://docs.rs/metrics/0.24.3/metrics/macro.counter.html#example

itzmanish · 2026-02-05T09:47:57Z

                    // Spawn a new task to handle the connection
                    tasks.push(async move {
+                        // Track connection duration - records histogram on drop
+                        let _timing_guard = TimingGuard::new("moq_relay_connection_duration_seconds");


are we measuring time taken to do the connection or the duration for which the connection was active?

This was intended to measure the duration of the entire connection, but I've actually rethought that and I'm going to remove it for now.

…acade Adds optional metrics support behind the 'metrics' feature flag. When disabled, all instrumentation compiles to zero-cost no-ops. ## Metrics (15 total) ### Counters (7) - moq_relay_connections_total: Total incoming connections - moq_relay_connection_errors_total{stage}: Connection failures - moq_relay_publishers_total: Total ANNOUNCE requests - moq_relay_announce_errors_total: Announce handling failures - moq_relay_subscribers_total: Total SUBSCRIBE requests - moq_relay_subscribe_failures_total{reason}: Subscription failures - moq_relay_upstream_errors_total{stage}: Upstream connection failures ### Gauges (6) - moq_relay_active_connections: Current client connections - moq_relay_active_publishers: Current publishers - moq_relay_active_subscriptions: Current subscriptions - moq_relay_active_tracks: Tracks being served - moq_relay_announced_namespaces: Registered namespaces - moq_relay_upstream_connections: Upstream/origin connections ### Histograms (2) - moq_relay_connection_duration_seconds: Connection lifetimes - moq_relay_subscribe_latency_seconds{source}: Subscription resolution time ## Implementation - GaugeGuard: RAII guard for gauge increment/decrement - TimingGuard: RAII guard for histogram duration recording - Both guards have #[must_use] to prevent accidental drops - All metrics prefixed with moq_relay_ for namespace safety Consumers install their own metrics::Recorder to route these to their preferred backend (Prometheus, StatsD, etc.).

…lidation Add --metrics-addr CLI flag and metrics-prometheus feature to enable direct Prometheus metrics exposure from the standalone relay binary. This allows testing and validation of metrics without the crique adapter layer. Changes: - Add metrics-exporter-prometheus optional dependency - Add metrics-prometheus feature (implies metrics feature) - Add --metrics-addr <addr> CLI flag to expose /metrics endpoint - Remove connection_duration_seconds histogram (not suitable for long-lived connections; use active_connections gauge instead) - Configure histogram buckets for subscribe_latency_seconds (1ms-10s) Usage: cargo build -p moq-relay-ietf --features metrics-prometheus ./moq-relay-ietf --metrics-addr 127.0.0.1:9090 ... curl http://127.0.0.1:9090/metrics

Add SessionError::is_graceful_close() method to detect when a connection ended due to a graceful APPLICATION_CLOSE (code 0) vs an actual error. Changes: - moq-transport: Add is_graceful_close() helper that checks error strings for graceful close patterns (transport-agnostic for future tokio-quiche) - moq-relay-ietf: Add connections_closed_total counter (always increments) - moq-relay-ietf: Only increment connection_errors_total for actual errors This allows dashboards/alerts to distinguish between healthy disconnects and error conditions. Previously all session closes were counted as errors.

- Fix is_graceful_close() for WebTransport connections - Add web-transport-quinn, web-transport-proto, and quinn dependencies - Use structural pattern matching instead of string matching - Properly decode HTTP/3 encoded WebTransport close codes - Add detailed comments about coupling to quinn backend - Fix subscribe failure double-counting - Split subscribe_failures_total into separate counters: - subscribe_not_found_total: track not found after checking all sources - subscribe_route_errors_total: infrastructure failure when routing - Route errors now return early with internal error instead of falling through to also count as not_found - Add route_error as a possible source label for subscribe latency - Fix connections_closed_total accounting gap - Increment connections_closed_total when session_accept fails - Maintains invariant: connections_total - connections_closed_total == active_connections

The metrics crate facade is effectively zero-cost when no recorder is installed (an atomic load + early return), similar to the log crate. Remove the 'metrics' feature flag and all 21 #[cfg(feature = "metrics")] annotations. The metrics-prometheus feature remains for the optional Prometheus exporter, which brings heavier dependencies. This simplifies every call site from: #[cfg(feature = "metrics")] metrics::counter!("...").increment(1); to just: metrics::counter!("...").increment(1); Also removes the ZST stub implementations of GaugeGuard and TimingGuard that were needed for the feature-disabled code path.

The upstream_connections gauge was incrementing before the QUIC connection and MoQ session were established. If either step failed, the gauge briefly spiked and then decremented — misrepresenting the number of active upstream connections. Move the GaugeGuard creation to after both connect() and Subscriber::connect() succeed so the gauge only reflects established upstream connections. Failed attempts are already captured by the upstream_errors_total counter.

Register metric descriptions with the recorder so they appear as # HELP comments in Prometheus output. Called after installing the exporter in main.rs.

- Format main.rs and relay.rs - Replace try_from().unwrap() with infallible from() in varint tests

nnazo

change LGTM after discussing the error modes for the coordinator errors 👍

englishm changed the title ~~feat(moq-relay-ietf): add metrics instrumentation via metrics crate facade~~ Feb 4, 2026

englishm-cloudflare force-pushed the observability-metrics branch 2 times, most recently from 2fede20 to e80d11e Compare February 4, 2026 23:47

itzmanish reviewed Feb 5, 2026

View reviewed changes

englishm-cloudflare force-pushed the observability-metrics branch from e80d11e to 86e877f Compare February 5, 2026 15:54

englishm-cloudflare marked this pull request as draft February 7, 2026 00:35

englishm added 4 commits February 7, 2026 19:55

englishm-cloudflare force-pushed the observability-metrics branch from f215397 to 31fd662 Compare February 8, 2026 03:15

englishm-cloudflare changed the title ~~Add metrics instrumentation via metrics crate facade~~ Feb 8, 2026

englishm-cloudflare marked this pull request as ready for review February 17, 2026 20:31

englishm added 2 commits February 17, 2026 15:38

feat(metrics): add describe_metrics() for Prometheus HELP text

b84ca35

Register metric descriptions with the recorder so they appear as # HELP comments in Prometheus output. Called after installing the exporter in main.rs.

fix: cargo fmt and clippy lints

8334d3d

- Format main.rs and relay.rs - Replace try_from().unwrap() with infallible from() in varint tests

nnazo approved these changes Feb 17, 2026

View reviewed changes

englishm merged commit 1206c2a into cloudflare:main Feb 18, 2026
1 check passed

Navigation Menu

Uh oh!

feat(moq-relay-ietf): add metrics instrumentation - #140

feat(moq-relay-ietf): add metrics instrumentation#140
englishm merged 8 commits into
cloudflare:mainfrom
englishm-cloudflare:observability-metrics

englishm-cloudflare commented Feb 4, 2026 •

edited

Loading

itzmanish Feb 5, 2026

englishm-cloudflare Feb 5, 2026

itzmanish Feb 5, 2026

englishm-cloudflare Feb 6, 2026

nnazo left a comment

Uh oh!

Labels

4 participants

Uh oh!

Conversation

englishm-cloudflare commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Metrics

Counters (10)

Gauges (6)

Histograms (1)

Implementation

Design Decisions

Usage

itzmanish Feb 5, 2026

Choose a reason for hiding this comment

englishm-cloudflare Feb 5, 2026

Choose a reason for hiding this comment

itzmanish Feb 5, 2026

Choose a reason for hiding this comment

englishm-cloudflare Feb 6, 2026

Choose a reason for hiding this comment

nnazo left a comment

Choose a reason for hiding this comment

Uh oh!

Labels

4 participants

englishm-cloudflare commented Feb 4, 2026 •

edited

Loading