Issue 3948: user cfg override fix

[ruslan-mikhailov]

What this PR does: Allows to put "span-metrics-size", "span-metrics-count", "span-metrics-latency" into user configurations overrides via API

Which issue(s) this PR fixes:
Fixes #3945

How it has been tested

Step 1. Run docker compose (here I used example/docker-compose/otel-collector-multitenant) with the following tempo.yaml:

overrides:
  defaults:
    metrics_generator:
      processors: [local-blocks] # leave only local-blocks
      generate_native_histograms: both
  user_configurable_overrides:
    enabled: true
    poll_interval: 5s 
    client:
      backend: local
      local:
        path: /var/tempo/overrides

Step 2. Check current configurations:

1. Set processors to "local-blocks" (the same as in configurations):

curl -X POST -v -H "If-Match: 1697726795401423" -H "X-Scope-OrgID: foo-bar-baz" 'http://localhost:3200/api/overrides?scope=merged' --data '{"metrics_generator":{"processors": ["local-blocks"]}}'

2. Get current configurations:

curl -X GET -v -H "X-Scope-OrgID: foo-bar-baz" 'http://localhost:3200/api/overrides?scope=merged'

Expected results: HTTP status code 200, metrics_generator.processors has value ["local-blocks"]
Actual results:

{"forwarders":null,"cost_attribution":{},"metrics_generator":{"processors":["local-blocks"],"disable_collection":false,"collection_interval":"0s","processor":{"service_graphs":{"dimensions":null,"enable_client_server_prefix":false,"peer_attributes":null,"histogram_buckets":null},"span_metrics":{"dimensions":null,"enable_target_info":false,"filter_policies":null,"histogram_buckets":null,"target_info_excluded_dimensions":null}}}}

Step 3. Check metrics:

Metric	Expected result
tempo_metrics_generator_processor_local_blocks_bytes	has data
traces_spanmetrics_calls_total	no data
traces_spanmetrics_latency	no data
traces_spanmetrics_size_total	no data

Step 4. Publish "local-blocks", "span-metrics-size", "span-metrics-count", "span-metrics-latency":

curl -X POST -v -H "If-Match: 1697726795401423" -H "X-Scope-OrgID: foo-bar-baz" 'http://localhost:3200/api/overrides?scope=merged' --data '{"metrics_generator":{"processors": ["local-blocks", "span-metrics-size", "span-metrics-count", "span-metrics-latency"]}}'

Expected results: HTTP status code 200, no errors in logs
Production image results: HTTP status code 400, body:

metrics_generator.processor "span-metrics-latency" is not a known processor, valid values: [service-graphs span-metrics local-blocks]

Step 5. Check resulted configurations:

curl -X GET -v -H "X-Scope-OrgID: foo-bar-baz" 'http://localhost:3200/api/overrides?scope=merged'

Expected results: metrics_generator.processors has value ["local-blocks", "span-metrics-size", "span-metrics-count", "span-metrics-latency"]
Actual results:

{"forwarders":null,"cost_attribution":{},"metrics_generator":{"processors":["span-metrics-size","span-metrics-count","span-metrics-latency","local-blocks"],"disable_collection":false,"collection_interval":"0s","processor":{"service_graphs":{"dimensions":null,"enable_client_server_prefix":false,"peer_attributes":null,"histogram_buckets":null},"span_metrics":{"dimensions":null,"enable_target_info":false,"filter_policies":null,"histogram_buckets":null,"target_info_excluded_dimensions":null}}}}

Step 6. Check metrics

Metric	Expected result
tempo_metrics_generator_processor_local_blocks_bytes	has data
traces_spanmetrics_calls_total	has data
traces_spanmetrics_latency	has data
traces_spanmetrics_size_total	has data

Checklist

• Tests updated
  (Tested manually)
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]