Discrepancy in kubelet_volume_stats_* Metrics Between EKS 1.31 (AL2) and EKS 1.33 (AL2023)

Hello AWS and EKS Experts,

We are experiencing a critical issue with metric availability after upgrading one of our EKS clusters and are seeking clarification on whether this is expected behavior.

Problem Summary

Our primary goal is to monitor PersistentVolumeClaim (PVC) usage per namespace using a standard Prometheus query that relies on kubelet_volume_stats_* metrics. This monitoring has been working perfectly in our production environment, but these metrics are completely missing after upgrading to EKS 1.33.

Environment Comparison

✅ Working Environment (Production)

EKS Version: 1.31

Node Groups: EKS Managed Node Group

AMI Type: AL2_x86_64

Observed Behavior: kubelet_volume_stats_* metrics are scraped successfully

CSI Drivers: EBS CSI, EFS CSI (both working)

❌ Non-Working Environment (After Upgrade)

EKS Version: 1.33

Node Groups: EKS Managed Node Group

AMI Type: AL2023_ARM_64_STANDARD

Observed Behavior: kubelet_volume_stats_* metrics completely absent from kubelet /metrics endpoint

CSI Drivers: Same configuration as working environment

This issue prevents us from using our standard PromQL query for PVC monitoring:

max by(namespace, persistentvolumeclaim) (
  (
    kubelet_volume_stats_capacity_bytes
    - kubelet_volume_stats_available_bytes
  ) / kubelet_volume_stats_capacity_bytes
) * 100

We have checked the official Kubernetes documentation (v1.33), which still lists kubelet_volume_stats_capacity_bytes, kubelet_volume_stats_available_bytes, etc., as available kubelet metrics. This creates confusion, as we would expect a standard Kubernetes metric to be available in a standard EKS environment.

Our assumption is that this change is related to the architectural differences between the Amazon Linux 2 (AL2) AMI and the newer Amazon Linux 2023 (AL2023) AMI, specifically regarding how cAdvisor is integrated in a pure CRI/CSI environment without dockershim.

Our Questions Are:

Is the absence of kubelet_volume_stats_* metrics on EKS nodes using the AL2023 AMI an expected and intentional change?

If this is intentional, could you confirm the underlying architectural reason? Is it due to the full adoption of the CSI storage model, which renders the old kubelet volume metric collection path obsolete?

What is the official, AWS-recommended method or PromQL query for monitoring PVC disk usage on modern EKS clusters (1.33+ with AL2023 AMIs)? We would appreciate official guidance.