[Cloud Security] [Findings Ingest Pipeline] Remove cloud.account.id and cloud.account.name empty fields from the latest findings document

[opauloh]

Summary

It fixes elastic/kibana#178904

This PR updates the Findings Ingest Pipeline to remove empty cloud.account.id and cloud.account.name fields in a Findings document. This fixes issues in Kibana when those fields are empty.

This PR also introduces Ingest Pipelines tests and adds tests to cover pipeline processors in a findings document. It can be tested following the documentation here

Screenshot

Pipeline Tests

Kibana Tests

Before changes:

Fields with empty cloud.account.name

Bug in group by cloud account visualization, result is hidden due to empty field #178904

After changes:

No more fields with empty cloud.account.name

Fields are instead missing (expected behaviour)

Group by cloud account visualization shows "No cloud account" (expected behaviour)

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[seanrathier]

Look good to me! Just a nit in the change log.

[seanrathier]

:aha moment
So the integration for cloud security posture handles the transformation of data collected from cloudbeat to the latest findings index.

:knowledge question
Is this rule in the ingestion pipeline removing the field from the document before it ends up the in the findings?
I am assuming this because the expected test results shows an empty object for the account

[opauloh]

So the integration for cloud security posture handles the transformation of data collected from cloudbeat to the latest findings index.

:knowledge question
Is this rule in the ingestion pipeline removing the field from the document before it ends up the in the findings?
I am assuming this because the expected test results shows an empty object for the account

Yes, and It's directly on the data stream index logs-cloud_security_posture.findings-* actually, (so it's before the transform runs to that aggregates the data into the logs-cloud_security_posture.findings_latest-* index.

The Ingest pipeline runs at the moment the data going to be inserted into the findings index

And how it's knows that it needs to run this pipeline for that index is defined with Index Templates, it's where the findings mappings are defined as well the default pipeline:

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 954402d

History

• 💚 Build #10159 succeeded 9428f15
• 💚 Build #9992 succeeded 1d8b2b0
• 💚 Build #9957 succeeded 5c61d84

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Package cloud_security_posture - 1.9.0-preview03 containing this change is available at https://epr.elastic.co/search?package=cloud_security_posture

[elasticmachine]

Package cloud_security_posture - 1.9.0 containing this change is available at https://epr.elastic.co/search?package=cloud_security_posture