[Azure][Storage_account] Enable TSDB

[ritalwar]

• Enhancement

Proposed commit message

Enable time series data streams for the storage_account metrics dataset. This dramatically reduces storage for metrics and is expected to progressively improve query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html.

The changes have been validated using the TSDB migration test kit.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.

• I have verified that all data streams collect metrics or logs.

• I have added an entry to my package's changelog.yml file.

• I have verified that Kibana version constraints are current according to guidelines.

• Closes Azure TSDB Enablement: storage_account datastream #7215

Screenshots

Refer #7215

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-10-03T06:39:17.168+0000

• Duration: 18 min 30 sec

Test stats 🧪

Test	Results
Failed	0
Passed	34
Skipped	0
Total	34

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (4/4)	💚
Files	100.0% (4/4)	💚
Classes	100.0% (4/4)	💚
Methods	75.758% (25/33)	👍
Lines	100.0% (32/32)	💚
Conditionals	100.0% (0/0)	💚

[lalit-satapathy]

Hi @zmoog, @ritalwar,

Do you see the storage_account data stream any impact from the Azure metrics grouping issues (example this) currently being worked on. or this one is not impacted by those changes?

[zmoog]

@lalit-satapathy, I guess the grouping / time grain problems can happen or not, depending on the metricset configuration.

If the migration kit didn't report any data loss, this probably means the storage account metrics config is not triggering the problem.

However, it's worth double-checking this metricset more in the context of the changes happening with #7621.

Thanks for bringing this up!

[ritalwar]

I conducted testing with the TSDB migration kit and confirmed that there was no data loss. Furthermore, I reviewed the metricbeat code and found that grouping for the storage account is already occurring using allowed dimensions, so there should be no impact.

In fact, for Azure storage accounts, we are also performing grouping based on the Timegrain. Please take a look.

[lalit-satapathy]

LGTM as this is not impacted by Azure metric grouping issue.

[elasticmachine]

Package azure_metrics - 1.2.0 containing this change is available at https://epr.elastic.co/search?package=azure_metrics