[Cloud Security] release Wiz version 2.0.0

[maxcold]

Proposed commit message

Changes:

• removing rule.references non-ecs as this field is not required anymore by Kibana https://github.com/elastic/security-team/issues/10793 . The field rule.reference should be used instead
• mappings for rule.* fields which were text changed to keyword to match ecs
• resource.id is provided from id when provider_id is missing on Wiz
• external: esc is added for ECS fields
• add cloud.account.name mapping to vulnerability cdr transform destination

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• relates to https://github.com/elastic/security-team/issues/10794
• fixes https://github.com/elastic/security-team/issues/10595
• fixes https://github.com/elastic/security-team/issues/10811
• fixes https://github.com/elastic/security-team/issues/10819
• fixes https://github.com/elastic/security-team/issues/10875

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package wiz 👍(2) 💚(1) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
vulnerability	3215.43	2557.54	-657.89 (-20.46%)	💔

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[andrewkroh]

The rule fields are part of ECS. Anywhere we have an ECS field we should depend on the ECS definition through the use of external: ecs to ensure that we always have consistent mappings. This ensures that there are never data type conflicts between data streams for ECS fields.

This is what it would look like to use external: ecs for the ECS fields in wiz -- d3d335c.

However, after doing that clean-up it becomes easier to spot other problems (not new in this PR)

1. There are two problematic field definitions that are conflicting with ECS.

   packages/wiz/data_stream/cloud_configuration_finding/fields/rule.yml:10:7 rule.description field declared as type text conflicts with the ECS data type keyword (conflict)
   packages/wiz/data_stream/cloud_configuration_finding/fields/rule.yml:14:7 rule.reference field declared as type text conflicts with the ECS data type keyword (conflict)
   

2. rule.remediation has multiple data types (keyword, text)

   packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/fields/rule.yml:12:7 keyword (conflict)
   packages/wiz/data_stream/cloud_configuration_finding/fields/rule.yml:12:7 text (conflict)
   

3. duplicate definitions of the same field in a data stream

   packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/fields/ecs.yml:1:3 event.created is declared 2 times (duplicate)
     packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/fields/ecs.yml:19:3 additional definition (duplicate)
   packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/fields/fields.yml:5:3 package.name is declared 2 times (duplicate)
     packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/fields/fields.yml:23:3 additional definition (duplicate)
   

[maxcold]

@andrewkroh thanks for the detailed feedback! I updated the mapping according to your suggestion. A couple of questions:

1. how do I check for conflicts with ECS? neither lint nor check produce the same output as you have in your comment
2. I changed rule.description and rule.reference to ecs and that means the mapping changed from text to keyword which is a breaking change as far as I understand. What should be the guidance for users in the changelog in that case?

[andrewkroh]

how do I check for conflicts with ECS?

I don't know that elastic-package has this capability yet. I have an unofficial means that I occasionally use to audit the mappings.

I changed rule.description and rule.reference to ecs and that means the mapping changed from text to keyword which is a breaking change as far as I understand. What should be the guidance for users in the changelog in that case?

It's hard to prescribe one solution because it depends on how the data is being used and what problem they encounter (types of queries or aggregations they are performing). One solution for conflicts mentioned in this blog post is to apply a runtime field to the old index to make the field a keyword.

[kcreddy]

Since we are adhering to ECS, should we even define the rule fields that are of external: ecs? The logs-* default ecs@mappings component template should take care of them for the datastream.

[maxcold]

@kcreddy @andrewkroh Looking at the value of the ecs@mappings in my stack I don't see it as a replacement of external: ecs . Actually, I would like to understand the use of ecs@mappings better, as in our team we understood that it is a replacement of manual mapping, but then not everything that is available in ECS also mapped in this component template. Eg. there are no mappings for the rule.* fields, except the ones falling into broader mappings such as *.name

{
  "dynamic_templates": [
    {
      "ecs_timestamp": {
        "mapping": {
          "ignore_malformed": false,
          "type": "date"
        },
        "match": "@timestamp"
      }
    },
    {
      "ecs_message_match_only_text": {
        "path_match": [
          "message",
          "*.message"
        ],
        "mapping": {
          "type": "match_only_text"
        },
        "unmatch_mapping_type": "object"
      }
    },
    {
      "ecs_non_indexed_keyword": {
        "path_match": [
          "*event.original"
        ],
        "mapping": {
          "index": false,
          "type": "keyword",
          "doc_values": false
        }
      }
    },
    {
      "ecs_non_indexed_long": {
        "path_match": [
          "*.x509.public_key_exponent"
        ],
        "mapping": {
          "index": false,
          "type": "long",
          "doc_values": false
        }
      }
    },
    {
      "ecs_ip": {
        "path_match": [
          "ip",
          "*.ip",
          "*_ip"
        ],
        "mapping": {
          "type": "ip"
        },
        "match_mapping_type": "string"
      }
    },
    {
      "ecs_wildcard": {
        "path_match": [
          "*.io.text",
          "*.message_id",
          "*registry.data.strings",
          "*url.path"
        ],
        "mapping": {
          "type": "wildcard"
        },
        "unmatch_mapping_type": "object"
      }
    },
    {
      "ecs_path_match_wildcard_and_match_only_text": {
        "path_match": [
          "*.body.content",
          "*url.full",
          "*url.original"
        ],
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "wildcard"
        },
        "unmatch_mapping_type": "object"
      }
    },
    {
      "ecs_match_wildcard_and_match_only_text": {
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "wildcard"
        },
        "unmatch_mapping_type": "object",
        "match": [
          "*command_line",
          "*stack_trace"
        ]
      }
    },
    {
      "ecs_path_match_keyword_and_match_only_text": {
        "path_match": [
          "*.title",
          "*.executable",
          "*.name",
          "*.working_directory",
          "*.full_name",
          "*file.path",
          "*file.target_path",
          "*os.full",
          "*email.subject",
          "*vulnerability.description",
          "*user_agent.original"
        ],
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "keyword"
        },
        "unmatch_mapping_type": "object"
      }
    },
    {
      "ecs_date": {
        "path_match": [
          "*.timestamp",
          "*_timestamp",
          "*.not_after",
          "*.not_before",
          "*.accessed",
          "created",
          "*.created",
          "*.installed",
          "*.creation_date",
          "*.ctime",
          "*.mtime",
          "ingested",
          "*.ingested",
          "*.start",
          "*.end",
          "*.indicator.first_seen",
          "*.indicator.last_seen",
          "*.indicator.modified_at",
          "*threat.enrichments.matched.occurred"
        ],
        "mapping": {
          "type": "date"
        },
        "unmatch_mapping_type": "object"
      }
    },
    {
      "ecs_path_match_float": {
        "path_match": [
          "*.score.*",
          "*_score*"
        ],
        "mapping": {
          "type": "float"
        },
        "path_unmatch": "*.version",
        "unmatch_mapping_type": "object"
      }
    },
    {
      "ecs_usage_double_scaled_float": {
        "path_match": "*.usage",
        "mapping": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "match_mapping_type": [
          "double",
          "long",
          "string"
        ]
      }
    },
    {
      "ecs_geo_point": {
        "path_match": [
          "*.geo.location"
        ],
        "mapping": {
          "type": "geo_point"
        }
      }
    },
    {
      "ecs_flattened": {
        "path_match": [
          "*structured_data",
          "*exports",
          "*imports"
        ],
        "mapping": {
          "type": "flattened"
        },
        "match_mapping_type": "object"
      }
    },
    {
      "all_strings_to_keywords": {
        "mapping": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "match_mapping_type": "string"
      }
    }
  ]
}

[kcreddy]

AFAIU the idea of ecs@mappings component template is to have a generic set of dynamic templates that works across all ECS entities, instead of defining mappings for specific entities like rule.*, vulnerability.* etc.

In our case, following are rule.* fields that are defined as external: ecs and the corresponding dynamic template inside ecs@mappings that is handling it:

Field in rule.yml	Handled by dynamic template
rule.uuid	all_strings_to_keywords
rule.id	all_strings_to_keywords
rule.name	ecs_path_match_keyword_and_match_only_text
rule.description	all_strings_to_keywords
rule.reference	all_strings_to_keywords

Comparing the above dynamic mappings with ECS rule fields, it matches correctly.

rule.remediation is anyway manually defined as it is not in ECS.

[maxcold]

@kcreddy now I see how keyword mapping is being solved in the template, thanks. Then yes, I agree, it makes sense to remove these explicit ecs mappings

[kcreddy]

Change external: esc to external: ecs in PR commit message.

[kcreddy]

@maxcold, should this PR also contain a documentation update suggesting users to update Initial Interval according to their needs, like we discussed? or is it going to be part of Cloud Security documentation and we just link it here?

Just asking as I need to possibly document same for AWS SecurityHub.

[maxcold]

@kcreddy regarding the docs, the work on them is in progress, when product with docs team come up with a final version we can incorporate these changes in our PRs

[efd6]

I don't think this is the correct way to do this; the versions that have been released should continue to exist.

Also, what is it that makes this a 2.0.0? What is breaking here?

[maxcold]

what would be the correct way to do that? From a user's point of view if they never enabled "show beta integrations", they would upgrade from 1.8.1 (latest released non-preview version) to 2.0.0 and would get all the changes from 1.9.0-preview* with the change, and as a user I would expect to see this as a part of 2.0.0 changelog

Also, what is it that makes this a 2.0.0? What is breaking here?

this is reflected in the changelog

    - description: Rely on external ecs for ESC fields. rule.reference, rule.descipriton and rule.remediation changed from text to keyword.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/11414
    - description: Remove rule.references field and its mapping. Please use the ECS rule.reference field instead.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/11414

[efd6]

The change here inserts a bunch of breaking changes before a bugfix that had already been released as a 1.9.0 preview. If correct chronological order had been used and they were added as a 1.9.0, the full set of 1.9.0 changes would not have been breaking; there would have been a single new version entry looking like this:

- version: "1.9.0"
  changes:
    - description: Add cloud.account.name mapping to latest_cdr_vulnerabilities transform destination.
      type: bugfix
      link: https://github.com/elastic/integrations/pull/<SOMEOTHERPRNUMBER>
    - description: Default to resource id when provider_id is missing for resource.id field in cloud_configuration_finding data stream.
      type: enhancement
      link: https://github.com/elastic/integrations/pull/<SOMEOTHERPRNUMBER>
    - description: Add cloud.account.name mapping to latest_cdr_vulnerabilities transform destination.
      type: enhancement
      link: https://github.com/elastic/integrations/pull/11414

(leaving all the other already published versions in tact)

then you could have added this breaking enhancement:

- version: "2.0.0"
  changes:
    - description: Rely on external ecs for ESC fields. rule.reference, rule.descipriton and rule.remediation changed from text to keyword.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/11414
    - description: Remove rule.references field and its mapping. Please use the ECS rule.reference field instead.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/11414

This is not an academic issue; there is a bugfix that is being hidden here from people who may not want to upgrade across a break, and also blocks them from getting non-breaking enhancements. Admittedly, there is a fix in 1.8.something hidden in a backport branch, so the bugfix is less of a concern, but the non-breaking enhancements are still an issue.

[maxcold]

The problem is that 1.9.0-preview1 and 1.9.0-preview3 introduce the latest transforms, which we plan to release as part of the CDR initiative. These changes require the 8.16.x version of the stack; that's why we were asked to hold them in an open PR until closer to the 8.16 release date, as the flow with preview is not advised to be used. So, I can't release 1.9.0 just yet. Additionally, there are problems that we found while testing the CDR approach, together with the product folks, which require breaking changes before we can release transforms. That's why I'm skipping the 1.9.0 release and moving to 2.0.0, as all the changes there are bundled under CDR. We don't want users to be able to install the version with the transforms before the breaking changes are made. I can move the breaking change entries to the top of the changelog if you think that this would help.

[efd6]

This is deeply dissatisfying. I have removed my block.

[kcreddy]

LGTM for my comments.
Please merge after Dan's comments are addressed regarding version.

Relented

[CohenIdo]

There are several fields configured as ecs that also have a specified type. Is there a reason for this configuration?

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 3f11a0f

History

• 💚 Build #17675 succeeded 93271df
• 💚 Build #17638 succeeded e7616b0
• 💔 Build #17620 failed 15734c8
• 💚 Build #17499 succeeded 1184a6b
• 💔 Build #17497 failed 1623967
• 💔 Build #17492 failed a2cbc19

[elastic-sonarqube]

Quality Gate failed

Failed conditions
65.6% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elastic-vault-github-plugin-prod]

Package wiz - 2.0.0 containing this change is available at https://epr.elastic.co/search?package=wiz