Skip to content

[Cloud Security] Update manifest format_version to 3.0.0 for cloud_security_posture package - #8162

Merged
maxcold merged 6 commits into
elastic:mainfrom
maxcold:update_cloud_security_posture_format_version_to_3
Oct 16, 2023
Merged

[Cloud Security] Update manifest format_version to 3.0.0 for cloud_security_posture package#8162
maxcold merged 6 commits into
elastic:mainfrom
maxcold:update_cloud_security_posture_format_version_to_3

Conversation

@maxcold

@maxcold maxcold commented Oct 11, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

making necessary changes to update cloud_security_posture package format_version to 3.0.0 to opt in it for serverless:

  • updated type: object field mappings either to more suitable types or added object_type as it's required now
  • added DLM data_retention of 180d for findings index to enable data retention in serverless as ILM is not supported in serverless due to the issue will be handled separately as a part of https://github.com/elastic/security-team/issues/7642
  • added security capability so the package shows up in serverless

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots
@maxcold maxcold added the enhancement New feature or request label Oct 11, 2023
@maxcold maxcold changed the title Update manifest format_version to 3.0.0 for cloud_security_posture package Oct 11, 2023
@maxcold maxcold added the Team:Cloud Security Cloud Security team [elastic/cloud-security-posture] label Oct 11, 2023
@elasticmachine

elasticmachine commented Oct 11, 2023 •
edited
Loading

Copy link
Copy Markdown

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-10-12T13:50:36.790+0000

  • Duration: 15 min 7 sec

Test stats 🧪

Test Results
Failed 0
Passed 4
Skipped 0
Total 4

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.
@elasticmachine

elasticmachine commented Oct 11, 2023 •
edited
Loading

Copy link
Copy Markdown

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (0/0) 💚
Files 100.0% (0/0) 💚
Classes 100.0% (0/0) 💚
Methods 25.0% (2/8) 👎 -70.506
Lines 100.0% (0/0) 💚 2.045
Conditionals 100.0% (0/0) 💚
@maxcold
maxcold marked this pull request as ready for review October 11, 2023 11:45
@maxcold
maxcold requested a review from a team as a code owner October 11, 2023 11:45
orestisfl
orestisfl reviewed Oct 12, 2023
View reviewed changes
Comment thread packages/cloud_security_posture/changelog.yml Outdated
@maxcold
maxcold requested a review from orestisfl October 12, 2023 11:56
@maxcold

maxcold commented Oct 12, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@orestisfl I found an issue with specifying data_retention for our data stream. When I try to install the package I get an error

"Internal Server Error","message":"illegal_argument_exception\n\tCaused by:\n\t\tillegal_argument_exception: index template [logs-cloud_security_posture.findings_latest] specifies lifecycle configuration that can only be used in combination with a data stream\n\tRoot causes:\n\t\tillegal_argument_exception: updating component template [logs-cloud_security_posture.findings@package] results in invalid composable template [logs-cloud_security_posture.findings_latest] after templates are merged"

I think this is because we have logs-cloud_security_posture.findings which is a data stream and logs-cloud_security_posture.findings_latest which is an index and they share the same index template as it seems. I think it needs more investigation on how to solve it properly.

To not block the format_version update I will remove the lifecycle addition from this PR and summarise the problem in https://github.com/elastic/security-team/issues/7642 to do that as a follow-up. Adding data_retention for our data stream is not a must for serverless, we should be able to do it independent of the version update
@Omolola-Akinleye
Omolola-Akinleye self-requested a review October 12, 2023 14:11
@maxcold

maxcold commented Oct 12, 2023

Copy link
Copy Markdown
Contributor Author

I tested the package by building it and installing it on the stack created via elastic-package. After that, I added CNVM and CSPM integrations to check if the data is ingested properly. Everything looks fine to me
orestisfl
orestisfl reviewed Oct 13, 2023
View reviewed changes
Comment thread packages/cloud_security_posture/data_stream/vulnerabilities/fields/cloud.yml
type: keyword
- name: Security.security_groups
type: object
type: nested

@orestisfl orestisfl Oct 13, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In your tests, have you verified that these fields work as expected with the new type?

@maxcold maxcold Oct 13, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I didn't find anything weird about them. They show up in the document, and flyout Table/JSON view works, but searching doesn't work with these fields as it doesn't work without the change either, I guess because they are not being indexed
opauloh
opauloh approved these changes Oct 13, 2023
View reviewed changes

@opauloh opauloh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just checked integrity using elastic-package check and tested Kibana works properly using elastic-package stack
@maxcold
maxcold merged commit befdc5c into elastic:main Oct 16, 2023
@maxcold
maxcold deleted the update_cloud_security_posture_format_version_to_3 branch October 16, 2023 14:06
@elasticmachine

elasticmachine commented Oct 16, 2023

Copy link
Copy Markdown

Package cloud_security_posture - 1.6.2 containing this change is available at https://epr.elastic.co/search?package=cloud_security_posture
@andrewkroh andrewkroh added the Integration:cloud_security_posture Security Posture Management label Jul 22, 2024
orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
@maxcold
[Cloud Security] Update manifest format_version to 3.0.0 for cloud_se…
a74c7f4 
…curity_posture package (elastic#8162)

* Update manifest format_version to 3.0.0 for cloud_security_posture package

* added Github PR link to changelog

* changed version to 1.6.1-preview1

* remove preivew suffix

* remove DLM definition from findings data_stream due to the conflict with findings_latest index
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:cloud_security_posture Security Posture Management Team:Cloud Security Cloud Security team [elastic/cloud-security-posture]

5 participants

@maxcold @elasticmachine @orestisfl @opauloh @andrewkroh