Allow rerouting of Azure Metrics events to a different data stream

[zmoog]

Proposed commit message

Add permissions to reroute metrics events in all data streams in Azure Metrics.

The additional permissions will allow us to route metrics events to use the Azure resource type as dataset to partition metrics.

Rerouting can be set up by defining routing rules at the integration level or allowing users to add a reroute processor to the custom ingest pipeline.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

1. Install the Azure Monitor integration
2. Add a custom pipeline

_simulate

Here's a sample custom pipeline:

# metrics-azure.monitor@custom
[
  {
    "reroute": {
      "dataset": [
        "whatever"
      ],
      "if": "ctx.network?.name == 'Whatever'"
    }
  }
]

Note: don't forget to set _index to a valid value like metrics-azure.monitor-default, or you'll get an error from the pipeline simulator.

Test with a document that DOES NOT match the reroute condition:

# request
POST _ingest/pipeline/metrics-azure.monitor-1.3.0/_simulate
{
  "docs": [
    {
      "_index": "metrics-azure.monitor-default",
      "_source": {
        "@timestamp": "2022-10-04T13:05:22.643+1300",
        "network": {
          "name": "Guest"
        }
      }
    }
  ]
}

# response
{
  "docs": [
    {
      "doc": {
        "_index": "metrics-azure.monitor-default",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "@timestamp": "2022-10-04T13:05:22.643+1300",
          "network": {
            "name": "Guest"
          }
        },
        "_ingest": {
          "timestamp": "2023-11-17T09:51:44.203447627Z"
        }
      }
    }
  ]
}

Test with a document that DOES match the reroute condition:

# request
POST _ingest/pipeline/metrics-azure.monitor-1.3.0/_simulate
{
  "docs": [
    {
      "_index": "metrics-azure.monitor-default",
      "_source": {
        "@timestamp": "2022-10-04T13:05:22.643+1300",
        "network": {
          "name": "Whatever"
        }
      }
    }
  ]
}

# response
{
  "docs": [
    {
      "doc": {
        "_index": "metrics-whatever-default",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "@timestamp": "2022-10-04T13:05:22.643+1300",
          "data_stream": {
            "namespace": "default",
            "type": "metrics",
            "dataset": "whatever"
          },
          "network": {
            "name": "Whatever"
          }
        },
        "_ingest": {
          "timestamp": "2023-11-17T09:51:08.329205971Z"
        }
      }
    }
  ]
}

Real documents

Here's test with an actual document.

Create a new document:

# request
POST metrics-azure.monitor-default/_doc
{
  "@timestamp": "2022-10-04T13:05:22.643+1300",
  "network": {
    "name": "Whatever"
  }
}

# response
{
  "_index": ".ds-metrics-whatever-default-2023.11.17-000001",
  "_id": "PlC33IsB1zkxCGc3V9mP",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 1,
  "_primary_term": 1
}

# request
GET metrics-whatever-default/_search

# response
{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-metrics-whatever-default-2023.11.17-000001",
        "_id": "C1C03IsB1zkxCGc3Ptks",
        "_score": 1,
        "_source": {
          "data_stream": {
            "namespace": "default",
            "type": "metrics",
            "dataset": "whatever"
          },
          "@timestamp": "2022-10-04T13:05:22.643+1300",
          "network": {
            "name": "Whatever"
          }
        }
      }
    ]
  }
}

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-11-15T21:33:10.825+0000

• Duration: 16 min 31 sec

Test stats 🧪

Test	Results
Failed	0
Passed	34
Skipped	0
Total	34

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (4/4)	💚
Files	100.0% (4/4)	💚
Classes	100.0% (4/4)	💚
Methods	75.758% (25/33)	❕
Lines	100.0% (32/32)	💚
Conditionals	100.0% (0/0)	💚

[kaiyan-sheng]

I need to do the same for aws 🙂

[zmoog]

Thank you all for the reviews! ❤️

I also added the "How to test this PR locally" with tests using the pipeline simulator and actual documents to provide a reference mini-test suite for the next PRs.

[elasticmachine]

Package azure_metrics - 1.3.0 containing this change is available at https://epr.elastic.co/search?package=azure_metrics