Loki 3.5.3 fails to parse Thanos object store configuration with "field not found in type aws.S3Config"

Summary

When migrating from legacy S3 storage to Thanos-based object storage in Loki 3.5.3 using the official Helm chart (v6.35.1), Loki fails to start because it attempts to parse Thanos object store configuration fields using the legacy aws.S3Config parser, despite use_thanos_objstore: true being correctly set.

Environment

• Loki Version: 3.5.3
• Helm Chart Version: loki-6.35.1 (from https://grafana.github.io/helm-charts)
• Kubernetes: 1.31.x
• Object Storage: Hetzner Object Storage (S3-compatible)
• Deployment Method: Helm via Flux CD

Configuration

Helm Values (Base Configuration)

loki:
  auth_enabled: false
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  storage:
    use_thanos_objstore: true
    bucketNames:
      chunks: idp-st-loki
      ruler: idp-st-loki-ruler
    object_store:
      s3:
        bucket_name: idp-st-loki
  storage_config:
    object_store:
      s3:
        bucket_name: idp-st-loki
  deploymentMode: SimpleScalable
  ruler:
    enabled: true
  # ... other standard config

SOPS Overlay (Credentials and Endpoint)

loki:
  storage:
    object_store:
      s3:
        access_key_id: "REDACTED"
        secret_access_key: "REDACTED"
        endpoint: "https://REDACTED"
        bucket_lookup_type: path
        dualstack_enabled: false
        insecure: false

Generated Configuration (Merged Result)

storage_config:
  use_thanos_objstore: true
  
common:
  storage:
    object_store:
      s3:
        access_key_id: "REDACTED"
        bucket_name: idp-st-loki
        bucket_lookup_type: path
        dualstack_enabled: false
        endpoint: https://REDACTED
        http: {}
        insecure: false
        region: null
        secret_access_key: "REDACTED"
        sse: {}
      storage_prefix: null

# Also generates ruler_storage with similar config structure

Error Messages

failed parsing config: /etc/loki/config/config.yaml: yaml: unmarshal errors:
  line 78: field bucket_lookup_type not found in type aws.S3Config
  line 79: field bucket_name not found in type aws.S3Config
  line 80: field dualstack_enabled not found in type aws.S3Config
  line 82: field http not found in type aws.S3Config
  line 87: field storage_prefix not found in type base.RuleStoreConfig

Analysis

The issue appears to be that despite use_thanos_objstore: true being correctly set in the storage_config section, Loki 3.5.3 is still attempting to parse Thanos object store configuration fields using the legacy aws.S3Config struct instead of switching to the Thanos object store configuration parser.

Root Cause

The configuration contains valid Thanos-specific fields that are being rejected by the legacy parser:

• bucket_lookup_type - Controls path vs virtual-hosted style URLs (Thanos field)
• bucket_name - Single bucket name (Thanos uses one bucket vs legacy multi-bucket)
• dualstack_enabled - IPv4/IPv6 dual-stack support (Thanos field)
• http - HTTP client configuration (Thanos field)
• storage_prefix - Object key prefix (Thanos field)

The error messages show these fields are not found in type aws.S3Config, indicating the legacy parser is being used when the Thanos parser should be active.

Expected Behavior

With use_thanos_objstore: true, Loki should:

1. Switch to the Thanos object store configuration parser
2. Accept Thanos-specific configuration fields like bucket_name, bucket_lookup_type
3. Successfully parse the object store configuration
4. Connect to S3-compatible storage using the provided configuration

Actual Behavior

Despite the flag being set, Loki continues using the legacy aws.S3Config parser which rejects Thanos-specific fields, causing startup failures. This suggests either:

1. The use_thanos_objstore flag is not properly switching parser contexts
2. Some configuration sections are still processed by legacy parsers regardless of the flag
3. There's a configuration structure issue preventing proper parser selection

Steps to Reproduce

1. Configure Loki with Thanos object store:

   helm template grafana-loki loki --version 6.35.1 \
     --repo https://grafana.github.io/helm-charts \
     --set loki.useTestSchema=true \
     --set loki.storage.use_thanos_objstore=true \
     --set loki.storage.bucketNames.chunks=test-bucket \                                                                                                                                         
     --set loki.storage.bucketNames.ruler=test-bucket-ruler \
     --set loki.storage.object_store.s3.bucket_name=test-bucket \
     --set loki.storage.object_store.s3.endpoint=https://s3.example.com \
     --set loki.storage.object_store.s3.bucket_lookup_type=path \
     --show-only templates/config.yaml

   Note: The old loki.storage.bucketNames properties are still required, even with Thanos object store being enabled.

2. Deploy the generated configuration:
   The Helm chart generates a valid-looking configuration with use_thanos_objstore: true

3. Observe startup failure:
   Loki pods fail with the field parsing errors mentioned above

Workaround

Currently using legacy S3 configuration with s3forcepathstyle: true for S3-compatible storage providers.

Additional Context

• Migration Context: After updating the Helm chart from 6.34 to 6.35, Loki no longer started due to changed handling of legacy s3 configuration. So it was a good motivation for attempting migrating from legacy S3 storage to Thanos-based storage as part of future-proofing efforts (legacy client deprecation)
• Use Case: Running Loki in SimpleScalable mode on Kubernetes with external S3-compatible object storage
• Infrastructure: Non-AWS S3-compatible storage (Hetzner), requiring path-style URLs

Questions

1. Is Thanos object store fully supported in Loki 3.5.3? The configuration generates without errors but runtime parsing fails
2. Are there additional configuration requirements beyond use_thanos_objstore: true?
3. Is this a regression or a configuration issue on our part?
4. What's the minimum Loki version with full Thanos object store support?

Debugging Information

Helm Chart Generation Success

The Helm template generation succeeds and produces valid-looking YAML, suggesting the chart supports the configuration structure.

Configuration Validation

# Template validation passes
helm template ... --show-only templates/config.yaml  # ✅ Success

# Runtime parsing fails  
kubectl logs loki-write-0  # ❌ Field parsing errors

Storage Config Section

storage_config:
  use_thanos_objstore: true  # ← This flag is correctly set
  # ... but Loki still uses legacy parser

This suggests a potential disconnect between the configuration flag and the actual parser being used at runtime.