[8.17](backport #41790) Concurrent fetch of azure metricdefinitions and batchApi usage

[mergify]

The changes affect azure monitor and relevant metricsets. The list of metricsets affected are:

• monitor
• container_registry
• container_instance
• container_service
• compute_vm
• compute_vm_scaleset
• database_account
• storage_account

A new configuration parameter is introduced enable_batch_api of type boolean.
If set to false(default) nothing changes in the way the metrics are collected for these metricsets.

If set to true:

• The metric definitions of resources are collected asynchronously and write the results in a channel.
• The channel is read and when the number of definitions collected reach 50 (batch API limit)
• The metrics definitions are grouped based on criteria(1) and the azure BatchAPI is used to retrieve
  metrics of multiple resources with one api call.

1. Grouping criteria are

• Namespace
• SubscriptionID
• Location
• Names
• TimeGrain
• Dimensions

Proposed commit message

• WHAT: Introduce enable_batch_api parameter for concurrent fetching of azure metric definitions and metric values collection using Batch Api
• WHY: Helps mitigating scalability problems

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Relates Improve Azure Monitor scalability and performance #38624

Use cases

Screenshots

Logs

---

This is an automatic backport of pull request #41790 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 13f8fde has failed:

On branch mergify/bp/8.17/pr-41790
Your branch is up to date with 'origin/8.17'.

You are currently cherry-picking commit 13f8fde9a.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   NOTICE.txt
	new file:   x-pack/metricbeat/module/azure/README.md
	modified:   x-pack/metricbeat/module/azure/_meta/docs.asciidoc
	modified:   x-pack/metricbeat/module/azure/azure.go
	modified:   x-pack/metricbeat/module/azure/client.go
	new file:   x-pack/metricbeat/module/azure/client_batch.go
	new file:   x-pack/metricbeat/module/azure/client_batch_test.go
	modified:   x-pack/metricbeat/module/azure/client_utils.go
	modified:   x-pack/metricbeat/module/azure/config.go
	modified:   x-pack/metricbeat/module/azure/data.go
	modified:   x-pack/metricbeat/module/azure/mock_service.go
	new file:   x-pack/metricbeat/module/azure/monitor/client_helper_concurrent.go
	new file:   x-pack/metricbeat/module/azure/monitor/client_helper_concurrent_test.go
	modified:   x-pack/metricbeat/module/azure/monitor/monitor.go
	modified:   x-pack/metricbeat/module/azure/monitor_service.go
	modified:   x-pack/metricbeat/module/azure/resources.go
	modified:   x-pack/metricbeat/module/azure/service_interface.go
	new file:   x-pack/metricbeat/module/azure/storage/client_helper_concurrent.go
	new file:   x-pack/metricbeat/module/azure/storage/client_helper_concurrent_test.go
	modified:   x-pack/metricbeat/module/azure/storage/storage.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   go.mod
	both modified:   go.sum
	both added:      metricbeat/docs/modules/azure.asciidoc

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[mergify]

This pull request has not been merged yet. Could you please review and merge it @zmoog, @MichaelKatsoulis? 🙏

[zmoog]

/test

[zmoog]

/test

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/8.17/pr-41790 upstream/mergify/bp/8.17/pr-41790
git merge upstream/8.17
git push upstream mergify/bp/8.17/pr-41790

[mergify]

This pull request has not been merged yet. Could you please review and merge it @zmoog, @MichaelKatsoulis? 🙏