[azure logs] use one input + routing for all data streams (WIP)

[zmoog]

Proposed commit message

Switch the integration package from the one-input-per-data-stream model to the one-input model.

One input per data stream model:

One input model:

In the one-input model, there is only one azure-eventhub input running and sending events to the events data stream. In the events data stream, the ingest pipeline performs these tasks:

• discover and set the event.dataset field using the category field in the event.
• use the event.dataset field to reroute the event to the target data stream.

The discover process uses the following logic:

• if the event doesn't have a category, it sets event.dataset to azure.eventhub (the generic integration)
• if the event does have a category, it sets event.dataset to azure.platformlogs (it's probably an Azure log)
• if the event category is supported, it sets event.dataset to specific one like azure.activitylogs or azure.signinlogs.

After the discovery step, the routing rules use the event.dataset value to forward the events to the best available target data stream.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes Azure Logs: use one input per agent policy #11251

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elastic-sonarqube]

Quality Gate failed

Failed conditions
26.1% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[zmoog]

Here's the latest iteration for the integration settings to evidence the recommended single data stream (v2) vs. the legacy multiple data stream (v1) solutions:

[defutek-tj]

@zmoog - do you have any updates as to the timeframe when this will be ready / released?

[zmoog]

@zmoog - do you have any updates as to the timeframe when this will be ready / released?

Hey @defutek-tj, my apologies for the delay.

We clarified the Event Hub SDK behavior with the Event Hub team, and now we can move forward with the first update.

In the following integration update, we will ship a new data stream v2 (preview) to handle all the log categories. ETA is next week.

[elastic-sonarqube]

Quality Gate failed

Failed conditions
26.2% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 45192fb7310a978faecaaebc2811f6b96d7eebe5

History

• 💚 Build #17218 succeeded 533db9d8ea33091ec782795284839d5c067110f3
• 💚 Build #17216 succeeded 63ffd49c03323132df00103228c1d8b7ad28fac6
• 💔 Build #17215 failed 75677cac39725f031d9a67df69c5ec87344c9804

cc @zmoog

[zmoog]

I'm replacing this PR with the simpler and backwards compatible #11984