[azure logs] Normalize capitalized Category field for proper data stream routing

[zmoog]

Proposed commit message

Rename invalid log category field Category as category to correctly infer the event.dataset, and perform the routing to the target data stream, if available.

Unfortunately, some Azure services produce logs with capitalized field names, violating the spec.

❌ Invalid format

{
  "Category": "ServicePrincipalSignInLogs",
  "CorrelationId": "ed598e34-c91b-41da-a89e-e5a97da4209d",
  "DurationMs": 0,
  ...
}

Instead, it should be something similar:

✅ Expected format

{
  "category": "ServicePrincipalSignInLogs",
  "correlationId": "ed598e34-c91b-41da-a89e-e5a97da4209d",
  "durationMs": 0,
  ...
}

To properly identify the event dataset and apply the correct routing rules, we need to convert them to lowercase.

Note: the target data stream is responsible for dealing with the capitalized field names. The events data stream's responsibility is to route to the target data stream, if available, or fall back to platform logs.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

cd packages/azure

$ elastic-package test pipeline -v -d events
--- Test results for package: azure - START ---
╭─────────┬─────────────┬───────────┬────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                                      │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ azure   │ events      │ pipeline  │ (ingest pipeline warnings test-invalid-category-field-raw.log) │ PASS   │ 649.352083ms │
│ azure   │ events      │ pipeline  │ test-invalid-category-field-raw.log                            │ PASS   │  40.554834ms │
╰─────────┴─────────────┴───────────┴────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: azure - END   ---
Done

Related issues

• TBA

[constanca-m]

Thank you so much @zmoog ! I thought we had to add Category support for everything, didn't even notice it was possible to just lowercase it.

[zmoog]

can't recall the details, but I skipped tests because I thought the reroute processor "can't be tested" properly, but it turns out it's definitely possible.

We can add a few test documents with the category|Category|CaTeGoRy field and check the data_stream.dataset we get as output.

I added a few tests for the invalid field Category.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[constanca-m]

Thank you Maurizio!

[zmoog]

Added one routing test for each supported log category:

$ elastic-package test pipeline -d events   
Run pipeline tests for the package
--- Test results for package: azure - START ---
╭─────────┬─────────────┬───────────┬────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                                      │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ azure   │ events      │ pipeline  │ (ingest pipeline warnings test-activitylogs-raw.log)           │ PASS   │ 558.513166ms │
│ azure   │ events      │ pipeline  │ (ingest pipeline warnings test-application_gateway-raw.log)    │ PASS   │ 482.429333ms │
│ azure   │ events      │ pipeline  │ (ingest pipeline warnings test-auditlogs-raw.log)              │ PASS   │ 475.755834ms │
│ azure   │ events      │ pipeline  │ (ingest pipeline warnings test-firewall_logs-raw.log)          │ PASS   │ 503.368542ms │
│ azure   │ events      │ pipeline  │ (ingest pipeline warnings test-graphactivitylogs-raw.log)      │ PASS   │ 488.667917ms │
│ azure   │ events      │ pipeline  │ (ingest pipeline warnings test-identity_protection-raw.log)    │ PASS   │ 512.766917ms │
│ azure   │ events      │ pipeline  │ (ingest pipeline warnings test-invalid-category-field-raw.log) │ PASS   │ 530.038041ms │
│ azure   │ events      │ pipeline  │ (ingest pipeline warnings test-provisioning-raw.log)           │ PASS   │ 493.391417ms │
│ azure   │ events      │ pipeline  │ (ingest pipeline warnings test-signinlogs-raw.log)             │ PASS   │ 511.469667ms │
│ azure   │ events      │ pipeline  │ (ingest pipeline warnings test-springcloudlogs-raw.log)        │ PASS   │ 558.499916ms │
│ azure   │ events      │ pipeline  │ test-activitylogs-raw.log                                      │ PASS   │  43.934875ms │
│ azure   │ events      │ pipeline  │ test-application_gateway-raw.log                               │ PASS   │  39.646875ms │
│ azure   │ events      │ pipeline  │ test-auditlogs-raw.log                                         │ PASS   │  38.612042ms │
│ azure   │ events      │ pipeline  │ test-firewall_logs-raw.log                                     │ PASS   │  41.445125ms │
│ azure   │ events      │ pipeline  │ test-graphactivitylogs-raw.log                                 │ PASS   │  38.820708ms │
│ azure   │ events      │ pipeline  │ test-identity_protection-raw.log                               │ PASS   │  38.927917ms │
│ azure   │ events      │ pipeline  │ test-invalid-category-field-raw.log                            │ PASS   │  38.571375ms │
│ azure   │ events      │ pipeline  │ test-provisioning-raw.log                                      │ PASS   │  39.498334ms │
│ azure   │ events      │ pipeline  │ test-signinlogs-raw.log                                        │ PASS   │  39.363542ms │
│ azure   │ events      │ pipeline  │ test-springcloudlogs-raw.log                                   │ PASS   │   40.66325ms │
╰─────────┴─────────────┴───────────┴────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: azure - END   ---
Done

[zmoog]

/test

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: b848c52

History

• 💔 Build #28296 failed 58393d5
• 💔 Build #28283 failed 58393d5
• 💔 Build #28281 failed 58393d5
• 💚 Build #28062 succeeded 9beea8d
• 💚 Build #28040 succeeded 607af7b

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
94.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elastic-vault-github-plugin-prod]

Package azure - 1.28.1 containing this change is available at https://epr.elastic.co/package/azure/1.28.1/