[azure] update routing for signinlogs and springcloudlogs

[mmahacek]

Proposed commit message

Fix for routing Azure signinlogs from Azure AD.

The "Windows Azure Active Directory" resource generates logs with the category MicrosoftServicePrincipalSignInLogs. When using the Azure Events v2 stream, these are handled as platform logs instead of as signinLogs. This PR adds routing to flag these events as the correct dataset.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Ingest data with a {"category":"MicrosoftServicePrincipalSignInLogs"} and verify the document ends up in the signinlogs dataset.

Related issues

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package azure 👍(2) 💚(5) 💔(5)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
signinlogs	2583.98	2004.01	-579.97 (-22.44%)	💔
springcloudlogs	8064.52	5464.48	-2600.04 (-32.24%)	💔
eventhub	500000	250000	-250000 (-50%)	💔
identity_protection	6666.67	4784.69	-1881.98 (-28.23%)	💔
platformlogs	5263.16	4273.5	-989.66 (-18.8%)	💔

To see the full report comment with /test benchmark fullreport

[zmoog]

Hey @mmahacek, thanks for contributing this update!

I wonder if we should consider sing-in logs all *SignInLogs log categories, @kaiyan-sheng WDYT?

[mmahacek]

The signinlogs pipeline does drop any non "*signinlogs" before processing, so this does make some sense.

https://github.com/elastic/integrations/blob/main/packages/azure/data_stream/signinlogs/elasticsearch/ingest_pipeline/default.yml#L46-L48

[mmahacek]

While I'm updating routing rules, I notice that the router for springcloudlogs has a few different values:

integrations/packages/azure/data_stream/events/elasticsearch/ingest_pipeline/default.yml

Lines 106 to 110 in 80b9687

	- set:
	field: event.dataset
	value: azure.springcloudlogs
	if: 'ctx.tmp_json?.category == "ApplicationConsole" || ctx.tmp_json?.category == "SystemLogs" || ctx.tmp_json?.category == "IngressLogs" || ctx.tmp_json?.category == "BuildLogs" || ctx.tmp_json?.category == "ContainerEventLogs"'
	description: 'Azure Spring Apps log categories (refs: https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-logs/microsoft-appplatform-spring-logs)'

But the pipeline for that dataset only keeps two of the values:

integrations/packages/azure/data_stream/springcloudlogs/elasticsearch/ingest_pipeline/default.yml

Lines 53 to 55 in 80b9687

	- drop:
	if: "ctx.azure.springcloudlogs.category != 'SystemLogs' && ctx.azure.springcloudlogs.category != 'ApplicationConsole'"
	ignore_failure: true

Should this be updated too?

[kaiyan-sheng]

I wonder if we should consider sing-in logs all *SignInLogs log categories, @kaiyan-sheng WDYT?

@zmoog Yeah I think that's probably safer for whatever new signin logs Azure is going to add!

[mmahacek]

😅 Another update came in for this integration. Pushing a version bump - should be no other functional changes.

[mmahacek]

@elastic/obs-ds-hosted-services This is ready for a code owners approval. Thanks in advance!

[rwestECI]

Any update here guys? My org is the one who reported this and needs it resolved.

[rwestECI]

Still waiting for this to be merged.

[btrieger]

#16538 which I have added adds another source type for a SignInLog. Can modify my PR after this is merged to only add the new fields.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 9d464fa

History

• 💔 Build #35583 failed 1079457
• 💚 Build #34640 succeeded 40c8ebb
• 💚 Build #33778 succeeded 215f2c2
• 💚 Build #33610 succeeded 3777a5a
• 💚 Build #33511 succeeded 9983786

cc @mmahacek

[elastic-vault-github-plugin-prod]

Package azure - 1.32.0 containing this change is available at https://epr.elastic.co/package/azure/1.32.0/