Refactor caching to support multiple independent caches

[joe-elliott]

What this PR does:
This PR refactors caches to allow for multiple independent caches that can be assigned to various roles. It lays the ground work for adding additional caches for things like parquet pages or frontend search jobs. This PR is completely backwards compatible, but does deprecate all the existing cache configuration fields in the store: config block.

• Replaces "shouldCache" on the backend calls with a "cacheInfo" struct that allows the cache layer to make its own decisions.
• Create a new top level "cacheProvider" module and config block (see below for an example). Different components can now request caches that fulfill different roles from the provider.
• Came with some nice cleanup that helped validate the refactor.
  • removing writerCallback func(*backend.BlockMeta, time.Time) from compaction
  • removing CacheControl from SearchOptions
  • deprecating CacheControl and replacing it with the idea of roles
• Previous config options are loaded into the cache provider by tempodb using the createLegacyCache() method.

Open Questions

• cache_min_compaction_level and cache_max_block_age were the only caching settings that were left alone. These are policy settings and I decided to leave them under storage.trace in the config block. Should we leave "policy" style settings in the normal config blocks? and only put caching settings in the cache provider? This is the approach I preferred but I thought it was worth discussing.

• Roles are currently configured as a yaml list, but I'm tempted to make them a pipe delimited list of roles. I think it would read better, but seems strange when yaml supports lists directly.

• Trace ID index had support for caching so I replicated it in this PR. Since this feature is known to be broken, I'm wondering if I should drop support until we invest the time to improve it.

New example config

cache:
  background_cache:
    writeback_goroutines: 5
  caches:
  - roles:
    - parquet-footer
    - parquet-column-idx
    - parquet-offset-idx
    memcached:
      host: memcached-instance
  - roles:
    - bloom
    redis:
      endpoint: redis-instance

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[knylander-grafana]

@joe-elliott We'll probably want to update the Upgrade page with information about this breaking change.

[mapno]

I like the changes. Only have a couple of comments.

cache_min_compaction_level and cache_max_block_age were the only caching settings that were left alone. These are policy settings and I decided to leave them under storage.trace in the config block. Should we leave "policy" style settings in the normal config blocks? and only put caching settings in the cache provider? This is the approach I preferred but I thought it was worth discussing.

I like it the way it is right now.

Roles are currently configured as a yaml list, but I'm tempted to make them a pipe delimited list of roles. I think it would read better, but seems strange when yaml supports lists directly.

I actually it reads better as a list hah. I wouldn't change it.

Trace ID index had support for caching so I replicated it in this PR. Since this feature is known to be broken, I'm wondering if I should drop support until we invest the time to improve it.

Yeah, I think we should do that. Or at least log something when that cache is enabled. It's not very well documented that that cache is not working.

[mapno]

LGTM