[AWS] Add the Security Hub data streams

[cla-checker-service]

💚 CLA has been signed

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-08-15T20:55:20.750+0000

• Duration: 36 min 26 sec

Test stats 🧪

Test	Results
Failed	0
Passed	162
Skipped	2
Total	164

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (13/13)	💚
Files	92.857% (13/14)	👎 -4.275
Classes	92.857% (13/14)	👎 -4.275
Methods	84.167% (202/240)	👎 -5.059
Lines	95.686% (5190/5424)	👍 4.944
Conditionals	100.0% (0/0)	💚

[flash1293]

It seems like one panel on the dashboard is using a non-existing field name (maybe it got renamed):

Also, is there a special reason most of the "event distribution" charts are pies, but some of them are bar charts?

As there are a lot of similar panels and the dashboard gets pretty long, would it make sense to split it up into multiple dashboards and link it up via a markdown visualization so the user can "select" the charts they are interested in instead of scrolling through a long dashboard? This really depends on how it's meant to be used though.

[andrewkroh]

There are multiple calls to now in here that need to match the already set X-Amz-Date header down to the second. This is a race condition. Instead of calling formatDate now "20060102T150405Z" twice, can you replace that with an access to the header value? Like (index .header "X-Amz-Date") assuming the header map is available in this context.

[andrewkroh]

@marc-gr @P1llus Have you come across this type of issue before? I was thinking it would useful to have a "now"-like function that returns the same time for all invocations within the context of a given request. This would allow all "now"s used in a request to be the same.

Or another option would to provide a generic scratch pad where you can store the current time once and then reference it later. A similar concept is used in consul-template https://github.com/hashicorp/consul-template/blob/main/docs/templating-language.md#scratch. The idea would be like

request.transforms:
  - set:
      target: scratch.RequestTime
      value: '[[ now ]]'
  - set:
      target: header.X-Amz-Date
      value: '[[formatDate (.scratch.RequestTime) "20060102T150405Z"]]'

[andrewkroh]

This does not fully fix the problem but it does reduce the number of now invocations to one (credit to @efd6). You can declare variable in the template then use the variable instead of calling now. Like [[$now := now]] then replace the now calls with $now.

[darshan-elastic]

Do we have any documentation for this?

[andrewkroh]

Assuming by "this" you mean declaring variables in templates, then you can look at https://pkg.go.dev/text/template#hdr-Variables. Variable declarations in templates is something built in to Go text/template.

[andrewkroh]

Leaving this comment here because it's close to the kibana.version. What are the Agent version requirements? Are there newly introduced httpjson template functions required for this?

[darshan-elastic]

Yes newly introduced httpjson template functions is required for this. So we have mentioned it in Readme that 8.4.0 version will be required.

[efd6]

It looks like this did not happen.

[andrewkroh]

For reviewers, I formatted this for to help make sense of it.

[[
  (sprintf
    "AWS4-HMAC-SHA256 Credential={{access_key_id}}/%s/{{aws_region}}/securityhub/aws4_request, SignedHeaders=host;x-amz-date, Signature=%s"
    (formatDate
      now
      "20060102"
    )
    (hmac
      "sha256"
      (hexDecode
        (hmac
          "sha256"
          (hexDecode
            (hmac
              "sha256"
              (hexDecode
                (hmac
                  "sha256"
                  (hexDecode
                    (hmac
                      "sha256"
                      "AWS4{{secret_access_key}}"
                      (formatDate
                        now
                        "20060102"
                      )
                    )
                  )
                  "{{aws_region}}"
                )
              )
              "securityhub"
            )
          )
          "aws4_request"
        )
      )
      "AWS4-HMAC-SHA256\n"
      (formatDate
        now
        "20060102T150405Z"
      )
      "\n"
      (sprintf
        "%s/%s\n"
        (formatDate
          now
          "20060102"
        )
        "{{aws_region}}/securityhub/aws4_request"
      )
      (hash
        "sha256"
        "POST\n"
        "/findings\n"
        "\n"
        "host:securityhub.{{aws_region}}.amazonaws.com\n"
        (sprintf
          "x-amz-date:%s\n\n"
          (formatDate
            now
            "20060102T150405Z"
          )
        )
        "host;x-amz-date\n"
        (hash
          "sha256"
          (sprintf
            "%s"
            .body
          )
        )
      )
    )
  )
]]

[andrewkroh]

I think this needs a system test so that we can test indexing. With only pipeline tests we can't catch various mapping issues.

[darshan-elastic]

/test

[darshan-elastic]

@andrewkroh We are facing below issue when we are updating version to 8.3.0

So can we do the changes in Redshift data stream?

[andrewkroh]

We are facing below issue when we are updating version to 8.3.0

Let's back out the 8.3.0 ECS change and handle the upgrade separately as to not mix the redshift updates with the security hub additions.

[darshan-elastic]

/test

[andrewkroh]

@elastic/obs-cloud-monitoring, can you please take a look at this PR? As a codeowner for aws you are a required reviewer.

[kaiyan-sheng]

looks good to me