Skip to content

[gcp] add support of parsing cluster notifications in audit data stream - #17830

Merged
brijesh-elastic merged 7 commits into
elastic:mainfrom
brijesh-elastic:gcp-2.47.2
Mar 19, 2026
Merged

[gcp] add support of parsing cluster notifications in audit data stream#17830
brijesh-elastic merged 7 commits into
elastic:mainfrom
brijesh-elastic:gcp-2.47.2

Conversation

@brijesh-elastic

@brijesh-elastic brijesh-elastic commented Mar 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

gcp: add support of parsing cluster notifications in audit data stream

This PR introduces support for parsing (GKE cluster notifications)[1]. Additionally, it expands
the `labels.payload` JSON string to ensure it is indexable for improved searchability.

This example is taken from the documentation and has been mocked for testing purposes.
It also bumps the minimum Kibana version to `8.18.0`, as the current stack version has a (bug)[2]
in the `user_agent processor`.

[1] https://docs.cloud.google.com/kubernetes-engine/docs/concepts/cluster-notifications 
[2] https://github.com/elastic/elasticsearch/issues/116950

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/gcp directory.
  • Run the following command to run tests.

elastic-package test -v

Related issues
@brijesh-elastic brijesh-elastic self-assigned this Mar 16, 2026
@brijesh-elastic
brijesh-elastic requested review from a team as code owners March 16, 2026 07:01
@brijesh-elastic brijesh-elastic added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:gcp Google Cloud Platform bugfix Pull request that fixes a bug issue Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Mar 16, 2026
@elasticmachine

elasticmachine commented Mar 16, 2026

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
@github-actions

github-actions Bot commented Mar 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

✅ Vale Linting Results

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 16, 2026

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport
@brijesh-elastic
brijesh-elastic requested a review from kcreddy March 16, 2026 07:58
@andrewkroh andrewkroh added Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Mar 16, 2026
kcreddy
kcreddy reviewed Mar 16, 2026
View reviewed changes
Comment thread packages/gcp/data_stream/audit/_dev/test/pipeline/test-cluster-notifications.json Outdated
Comment thread packages/gcp/docs/audit.md Outdated
Comment thread packages/gcp/changelog.yml Outdated
@brijesh-elastic
gcp: address review feedback for audit notifications
60fc47a 
- Remove agent-provided input.type from pipeline fixture
- Map notification labels to cloud.project.id and orchestrator.*
- Add gcp.audit.notification field description and mark changelog as enhancement
@brijesh-elastic
ran system test and fix changelog
59961ba
@brijesh-elastic brijesh-elastic changed the title [gcp] Fix ingest pipeline to parse cluster notifications in audit data stream Mar 17, 2026
@brijesh-elastic
brijesh-elastic requested a review from kcreddy March 17, 2026 11:22
kcreddy
kcreddy approved these changes Mar 17, 2026
View reviewed changes

@kcreddy kcreddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
gizas
gizas reviewed Mar 17, 2026
View reviewed changes
Comment thread packages/gcp/data_stream/audit/_dev/test/pipeline/test-audit.log-expected.json Outdated
"version": "10.15"
},
"version": "71.0"
"version": "71.0."

@gizas gizas Mar 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think that this changed in your pr but it puzzled me a bit. WDYT? why we have an extra . in the end?

@gizas gizas Mar 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

integrations/packages/netskope/data_stream/events/_dev/test/pipeline/test-events.log-expected.json

Line 538 in 5e19135

"version": "95.0."

@brijesh-elastic brijesh-elastic Mar 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ve ran pipeline testing on Elastic Stack v8.13.0 (minimum supported version for gcp integration currently). This version is affected by the user_agent processor bug referenced here.

@kcreddy, should we consider bumping the minimum required stack version to avoid this issue?

@kcreddy kcreddy Mar 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we consider bumping the minimum required stack version to avoid this issue?

I think it is safe enough since 8.18 has been out more than a year. We should go ahead.

@kcreddy kcreddy Mar 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update Proposed commit message accordingly before merging
@elasticmachine

elasticmachine commented Mar 19, 2026

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @brijesh-elastic
@brijesh-elastic
brijesh-elastic merged commit b7633a7 into elastic:main Mar 19, 2026
9 checks passed
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 19, 2026

Copy link
Copy Markdown
Contributor

Package gcp - 2.48.0 containing this change is available at https://epr.elastic.co/package/gcp/2.48.0/
@github-actions github-actions Bot mentioned this pull request Mar 19, 2026
Open
srilumpa pushed a commit to srilumpa/elastic-integrations that referenced this pull request Mar 23, 2026
@brijesh-elastic @srilumpa
gcp: add support of parsing cluster notifications in audit data stream (
93a0ce8 
elastic#17830)

This PR introduces support for parsing (GKE cluster notifications)[1]. Additionally, it expands
the `labels.payload` JSON string to ensure it is indexable for improved searchability.

This example is taken from the documentation and has been mocked for testing purposes.
It also bumps the minimum Kibana version to `8.18.0`, as the current stack version has a (bug)[2]
in the `user_agent processor`.

[1] https://docs.cloud.google.com/kubernetes-engine/docs/concepts/cluster-notifications 
[2] elastic/elasticsearch#116950
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes a bug issue documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:gcp Google Cloud Platform Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

6 participants

@brijesh-elastic @elasticmachine @gizas @kcreddy @ishleenk17 @andrewkroh