Fix `durationMs` parsing in auditlogs data stream by gregorywychowaniec-zt · Pull Request #16674 · elastic/integrations

gregorywychowaniec-zt · 2025-12-23T08:58:18Z

Proposed commit message

Same as encoutered in #15976, we got some auditlogs with durationMs as a string that break the pipeline. So same fix for auditlogs here :

Convert azure.auditlogs.durationMs as long type if the value comes as string

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

POST _ingest/pipeline/logs-azure.auditlogs-1.32.1/_simulate
{
"docs": [
{
"_source": {
"message": """
{ "time": "2022-01-22T18:15:02.3875429Z", "resourceId": "/tenants/4bbb79f7-5724-4c9e-95f3-de075f6ec090/providers/Microsoft.aadiam", "operationName": "Update service principal", "operationVersion": "1.0", "category": "AuditLogs", "tenantId": "4bbb79f7-5724-4c9e-95f3-de075f6ec090", "resultSignature": "None", "durationMs": "0", "callerIpAddress": "1.128.3.4", "correlationId": "87979703-118b-498f-99c2-ccd1a56f1a5a", "identity": "Managed Service Identity", "Level": 4, "properties": {"id":"Directory_87979703-118b-498f-99c2-ccd1a56f1a5a_ULAYA_144938566","category":"ApplicationManagement","correlationId":"87979703-118b-498f-99c2-ccd1a56f1a5a","result":"success","resultReason":"","activityDisplayName":"Update service principal","activityDateTime":"2022-01-22T18:15:02.3875429+00:00","loggedByService":"Core Directory","operationType":"Update","userAgent":null,"initiatedBy":{"app":{"appId":null,"displayName":"Managed Service Identity","servicePrincipalId":"b9814691-9ca1-4e55-a1ac-8ef5dd010ec0","servicePrincipalName":null}},"targetResources":[{"id":"a7d5dcbe-0627-4ddf-a2f4-86b6785bcc42","displayName":"billing-test-wus","type":"ServicePrincipal","modifiedProperties":[{"displayName":"TargetId.ServicePrincipalNames","oldValue":null,"newValue":"\"a70a7931-c387-4dce-9f35-fbf95bdcc91e;https://identity.azure.net/N8CUySpCeRFU3iB/PEuFlON4zd8+n8d3qgzrF1MviSY=\""}],"administrativeUnits":[]}],"additionalDetails":[{"key":"User-Agent","value":"Microsoft Azure Graph Client Library 2.1.17-internal"},{"key":"AppId","value":"a70a7931-c387-4dce-9f35-fbf95bdcc91e"}]}}
"""
}
}
]
}

cla-checker-service · 2025-12-23T08:58:22Z

💚 CLA has been signed

zmoog · 2025-12-23T09:23:41Z

Hey @gregorywychowaniec-zt, thanks for raising this PR! 🙇

I'll open an issue/PR to fix this field across all Azure integrations.

zmoog

LGTM, thanks for fixing this @gregorywychowaniec-zt.

elasticmachine · 2025-12-23T09:27:03Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

zmoog · 2025-12-23T09:50:45Z

I created the issue #16677 to normalize durationMs to long across Azure integrations.

Kavindu-Dodan · 2025-12-23T17:10:23Z

@gregorywychowaniec-zt I have resolved the merge conflict and with current approvals, this should be good to go

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan · 2025-12-29T20:54:25Z

/test

elastic-vault-github-plugin-prod · 2025-12-29T21:14:35Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2025-12-29T21:14:39Z

💚 Build Succeeded

Buildkite Build
Commit: cc73537

History

💔 Build #35956 failed cc05200

cc @zmoog

elastic-vault-github-plugin-prod · 2025-12-29T21:32:06Z

Package azure - 1.34.1 containing this change is available at https://epr.elastic.co/package/azure/1.34.1/

…ity_protection, provisioning, springcloudlogs Some Azure services emit durationMs as a string instead of a number. The auditlogs data stream already guards against this (fixed in elastic#16674); platformlogs was unaffected from the start. The other five data streams still did a plain rename of durationMs into event.duration, so a string value passed through unconverted: the nanosecond-multiplication script right after it fails silently (ignore_failure: true) for a String operand, leaving event.duration as a raw millisecond string instead of a long-typed nanosecond value. This causes ES mapping conflicts on event.duration. Applied the same convert-if-string / rename-if-not-string / remove pattern already used in auditlogs to all five remaining data streams. Added a pipeline test per data stream reproducing durationMs as a string, verified end-to-end against a live Elasticsearch instance via `elastic-package test pipeline --generate`: - activitylogs: "0" -> event.duration = 0 - eventhub: "0" -> event.duration = 0 - identity_protection: "0" -> event.duration = 0 - provisioning: "828" -> event.duration = 828000000 - springcloudlogs: "12" -> event.duration = 12000000 All 40 existing and new pipeline tests pass (elastic-package test pipeline). Fixes elastic#16677 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01STTaWPq6nDxykUYmFe73D3

…ity_protection, provisioning, springcloudlogs Some Azure services emit durationMs as a string instead of a number. The auditlogs data stream already guards against this (fixed in elastic#16674); platformlogs was unaffected from the start. The other five data streams still did a plain rename of durationMs into event.duration, so a string value passed through unconverted: the nanosecond-multiplication script right after it fails silently (ignore_failure: true) for a String operand, leaving event.duration as a raw millisecond string instead of a long-typed nanosecond value. This causes ES mapping conflicts on event.duration. Applied the same convert-if-string / rename-if-not-string / remove pattern already used in auditlogs to all five remaining data streams, including matching processor tags (missing on the original auditlogs processors, added here per current pipeline conventions). Added a pipeline test per data stream reproducing durationMs as a string, verified end-to-end against a live Elasticsearch instance via `elastic-package test pipeline --generate`: - activitylogs: "0" -> event.duration = 0 - eventhub: "50" -> event.duration = 50000000 - identity_protection: "0" -> event.duration = 0 - provisioning: "828" -> event.duration = 828000000 - springcloudlogs: "12" -> event.duration = 12000000 The eventhub fixture required an explicit `tags: [parse_message]` config, since default.yml only routes into parsed-message.yml (the file this fix touches) when that tag is set -- without it, the test would have silently passed without exercising the fix at all. Fixture filenames follow the `test-<package>-<datastream>-<type>-sample.log` convention; the identity_protection fixture omits the underscore (`identityprotection`) since package-spec rejects underscores in this file name segment. Verified with a full `elastic-package check` (lint + build) and the complete pipeline test suite for the whole azure package (not just the 5 touched data streams) in a clean, non-worktree clone -- all green, no regressions. Fixes elastic#16677 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01STTaWPq6nDxykUYmFe73D3

Convert durationMs to long when it's a string

80ea08b

gregorywychowaniec-zt requested review from a team as code owners December 23, 2025 08:58

gregorywychowaniec-zt added 2 commits December 23, 2025 10:01

Update changelog

d54c730

Update changelog

2d61601

zmoog approved these changes Dec 23, 2025

View reviewed changes

zmoog added Integration:azure Azure Logs Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Dec 23, 2025

zmoog added the bugfix Pull request that fixes a bug issue label Dec 23, 2025

zmoog self-assigned this Dec 23, 2025

kcreddy approved these changes Dec 23, 2025

View reviewed changes

ShourieG reviewed Dec 23, 2025

View reviewed changes

Comment thread packages/azure/data_stream/auditlogs/_dev/test/pipeline/test-audit-logs-duration-as-string.log Outdated

Comment thread packages/azure/data_stream/auditlogs/elasticsearch/ingest_pipeline/default.yml Outdated

Add newline at end of file

fa30753

ShourieG approved these changes Dec 23, 2025

View reviewed changes

Kavindu-Dodan approved these changes Dec 23, 2025

View reviewed changes

Merge branch 'main' into main

8f2f1eb

devamanv approved these changes Dec 24, 2025

View reviewed changes

Kavindu-Dodan and others added 2 commits December 29, 2025 10:20

Merge branch 'main' into main

58cdaa6

fix version and changelog

cc05200

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan enabled auto-merge (squash) December 29, 2025 18:27

fix format

cc73537

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan merged commit dbbf43f into elastic:main Dec 29, 2025
8 checks passed

smnschndr mentioned this pull request Jul 28, 2026

[azure] Normalize durationMs to long in activitylogs, eventhub, identity_protection, provisioning, springcloudlogs #20385

Open

Navigation Menu

Uh oh!

Fix `durationMs` parsing in auditlogs data stream - #16674

Fix `durationMs` parsing in auditlogs data stream#16674
Kavindu-Dodan merged 8 commits into
elastic:mainfrom
gregorywychowaniec-zt:main

gregorywychowaniec-zt commented Dec 23, 2025 •

edited

Loading

cla-checker-service Bot commented Dec 23, 2025 •

edited

Loading

zmoog commented Dec 23, 2025

zmoog left a comment

elasticmachine commented Dec 23, 2025

Uh oh!

Uh oh!

zmoog commented Dec 23, 2025

Kavindu-Dodan commented Dec 23, 2025

Kavindu-Dodan commented Dec 29, 2025

elastic-vault-github-plugin-prod Bot commented Dec 29, 2025

Uh oh!

elasticmachine commented Dec 29, 2025

elastic-vault-github-plugin-prod Bot commented Dec 29, 2025

Labels

7 participants

Uh oh!

Conversation

gregorywychowaniec-zt commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

cla-checker-service Bot commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

zmoog commented Dec 23, 2025

zmoog left a comment

Choose a reason for hiding this comment

elasticmachine commented Dec 23, 2025

Uh oh!

Uh oh!

zmoog commented Dec 23, 2025

Kavindu-Dodan commented Dec 23, 2025

Kavindu-Dodan commented Dec 29, 2025

elastic-vault-github-plugin-prod Bot commented Dec 29, 2025

🚀 Benchmarks report

Uh oh!

elasticmachine commented Dec 29, 2025

💚 Build Succeeded

History

elastic-vault-github-plugin-prod Bot commented Dec 29, 2025

Labels

7 participants

gregorywychowaniec-zt commented Dec 23, 2025 •

edited

Loading

cla-checker-service Bot commented Dec 23, 2025 •

edited

Loading