feat: ecs event spec by olegsu · Pull Request #346 · elastic/cloudbeat

olegsu · 2022-08-17T10:32:37Z

Related to https://github.com/elastic/security-team/issues/3782

cla-checker-service · 2022-08-17T10:32:40Z

💚 CLA has been signed

mergify · 2022-08-17T10:32:45Z

This pull request does not have a backport label. Could you fix it @olegsu? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit
NOTE: backport-skip has been added to this pull request.

eyalkraft

Good job!
small changes required

eyalkraft · 2022-08-17T11:43:43Z

+
+// Event According to https://www.elastic.co/guide/en/ecs/current/ecs-event.html
+// event.ingested property is not part of this struct as the fleet server setting it
+type Event struct {


Nice!
Please make sure we use beat's implementation if possible

Great, I was looking this one

@eyalkraft the ecs.Event like other ECS types is using only ecs custom struct type.
So the final document looks like this (with uppercase keys)
If you are familiar with the decoder of those ecs tags it would be great.
Otherwise, I suggest keeping the previous, custom sturct and to handle this later.

eyalkraft · 2022-08-17T11:54:01Z

@@ -84,7 +85,7 @@ func (d *Data) fetchIteration(ctx context.Context) {
 	start := time.Now()

 	cycleId, _ := uuid.NewV4()


There are some cycleId leftovers

eyalkraft · 2022-08-17T11:59:41Z

+		ID:       id.String(),
+		Kind:     ecs.KindState,
+		Sequence: seq,
+		Outcome:  ecs.OutcomeSuccess,


I now wonder If we should also send the failed evaluations as failed events...
I'll comment in the issue

Sounds as a good KPI

eyalkraft · 2022-08-17T12:12:43Z

+	"github.com/stretchr/testify/assert"
+)
+
+func TestOpaEvaluator_decode(t *testing.T) {


eyalkraft · 2022-08-17T12:13:11Z

 	}

 	err = decoder.Decode(result)
+	opaResult.Metadata.CreatedAt = now()


Since it's only used once that could maybe be

Suggested change

opaResult.Metadata.CreatedAt = now()

opaResult.Metadata.CreatedAt = time.Now().UTC()

The reason for this was to have the ability to overwrite the value to a predictable one in the test

eyalkraft

Looks good!

olegsu · 2022-08-18T14:53:08Z

A quick update following #346 (comment)

To support the ecs struct tags properly, we need to add to all the structs that we use ecs tag. Later, implement the decoder that knows how to convert the struct into mapstr.M as required in the beat.Event (https://github.com/elastic/beats/blob/main/libbeat/beat/event.go#L44)
Similar to what is been done by packetbeat

The migration to ecs tags can be done just after this will be merged as it mostly a refactor without any changes in terms of the structure of the outgoing event.

To close, the Event struct that was removed will be part of this PR

@eyalkraft @uri-weisman

Slack

eyalkraft · 2022-08-22T12:18:42Z

Thanks @olegsu! This PR looks great.
Can you please open a follow-up ticket regarding using the ecs tag and implementing the decoder, including the information from your last comment and including what is the motivation for us to do so? (pros vs cons)
Thanks!

gurevichdmitry

@olegsu, utils.py file approved. LGTM

mergify · 2022-08-29T13:04:08Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b ecs-event-spec upstream/ecs-event-spec
git merge upstream/main
git push upstream ecs-event-spec

https://www.elastic.co/guide/en/ecs/current/ecs-event.html Signed-off-by: Oleg Sucharevich <oleg.sucharevich@elastic.co>

Signed-off-by: Oleg Sucharevich <oleg.sucharevich@elastic.co>

github-actions · 2022-08-30T08:41:32Z

Cloudbeat CI 🤖

Allure Report: http://csp-allure-reports.s3.amazonaws.com/allure_reports/cloudbeat/prs/346/index.html

olegsu · 2022-09-11T14:20:02Z

Thanks @olegsu! This PR looks great. Can you please open a follow-up ticket regarding using the ecs tag and implementing the decoder, including the information from your last comment and including what is the motivation for us to do so? (pros vs cons) Thanks!

Done @eyalkraft
https://github.com/orgs/elastic/projects/705/views/47?filterQuery=iteration%3A%22sprint+15%22

olegsu requested a review from a team as a code owner August 17, 2022 10:32

mergify Bot assigned olegsu Aug 17, 2022

mergify Bot added the backport-skip label Aug 17, 2022

mergify Bot mentioned this pull request Aug 17, 2022

feat: ecs event.* #345

Closed

olegsu force-pushed the ecs-event-spec branch 4 times, most recently from 054ced0 to 96ef744 Compare August 17, 2022 11:49

eyalkraft suggested changes Aug 17, 2022

View reviewed changes

olegsu mentioned this pull request Aug 17, 2022

Update CSP mapping fields elastic/integrations#4016

Merged

4 tasks

olegsu force-pushed the ecs-event-spec branch from 96ef744 to 38c5226 Compare August 17, 2022 12:34

eyalkraft approved these changes Aug 17, 2022

View reviewed changes

olegsu mentioned this pull request Aug 17, 2022

[Cloud Posture] Add ecs event to csp types elastic/kibana#139015

Merged

olegsu force-pushed the ecs-event-spec branch 3 times, most recently from 080b0f3 to 2a2dddc Compare August 22, 2022 04:51

olegsu force-pushed the ecs-event-spec branch 3 times, most recently from cf6480f to 77c6934 Compare August 28, 2022 07:13

olegsu requested a review from a team as a code owner August 28, 2022 07:13

olegsu requested a review from gurevichdmitry August 28, 2022 07:13

gurevichdmitry approved these changes Aug 28, 2022

View reviewed changes

olegsu added 2 commits August 30, 2022 10:37

feat: add event matching to ecs spec

bf5129f

https://www.elastic.co/guide/en/ecs/current/ecs-event.html Signed-off-by: Oleg Sucharevich <oleg.sucharevich@elastic.co>

feat: remove cycle_id

0bc3b47

Signed-off-by: Oleg Sucharevich <oleg.sucharevich@elastic.co>

olegsu force-pushed the ecs-event-spec branch from 77c6934 to 0bc3b47 Compare August 30, 2022 07:37

olegsu merged commit 6af24a7 into elastic:main Aug 30, 2022

Navigation Menu

Uh oh!

feat: ecs event spec - #346

feat: ecs event spec#346
olegsu merged 2 commits into
elastic:mainfrom
olegsu:ecs-event-spec

olegsu commented Aug 17, 2022 •

edited

Loading

cla-checker-service Bot commented Aug 17, 2022 •

edited

Loading

mergify Bot commented Aug 17, 2022

eyalkraft left a comment

eyalkraft Aug 17, 2022

olegsu Aug 17, 2022

olegsu Aug 17, 2022

eyalkraft Aug 17, 2022

eyalkraft Aug 17, 2022

olegsu Aug 17, 2022

eyalkraft Aug 17, 2022

eyalkraft Aug 17, 2022

olegsu Aug 17, 2022

eyalkraft left a comment

olegsu commented Aug 18, 2022 •

edited

Loading

eyalkraft commented Aug 22, 2022

gurevichdmitry left a comment

mergify Bot commented Aug 29, 2022

github-actions Bot commented Aug 30, 2022

olegsu commented Sep 11, 2022 •

edited

Loading

Labels

3 participants

		@@ -84,7 +85,7 @@ func (d *Data) fetchIteration(ctx context.Context) {
		start := time.Now()

		cycleId, _ := uuid.NewV4()

	opaResult.Metadata.CreatedAt = now()
	opaResult.Metadata.CreatedAt = time.Now().UTC()

Uh oh!

Conversation

olegsu commented Aug 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

cla-checker-service Bot commented Aug 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mergify Bot commented Aug 17, 2022

eyalkraft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eyalkraft left a comment

Choose a reason for hiding this comment

olegsu commented Aug 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

eyalkraft commented Aug 22, 2022

gurevichdmitry left a comment

Choose a reason for hiding this comment

mergify Bot commented Aug 29, 2022

github-actions Bot commented Aug 30, 2022

Cloudbeat CI 🤖

olegsu commented Sep 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Labels

3 participants

olegsu commented Aug 17, 2022 •

edited

Loading

cla-checker-service Bot commented Aug 17, 2022 •

edited

Loading

olegsu commented Aug 18, 2022 •

edited

Loading

olegsu commented Sep 11, 2022 •

edited

Loading