Add datastream to collect audit logs in kubernetes integration

[tetianakravchenko]

Signed-off-by: Tetiana Kravchenko tetiana.kravchenko@elastic.co

What does this PR do?

This PR adds a new data stream in kubernetes integration to collect audit logs.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

cd  package/kubernetes
elastic-package build
elastic-package stack up -d -v --version 8.1.0-f479a40a-SNAPSHOT

2. Start kind with auditing enabled, as described here:

kind create cluster --config kind-config.yaml

3. connect networks:

docker network connect elastic-package-stack_default kind-control-plane

4. deploy elastic-agent:

kubectl apply -f elastic-agent-managed-kubernetes.yaml

Related issues

• Closes Add audit logs datastream in K8s integration #2285

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-12-15T11:20:17.386+0000

• Duration: 29 min 15 sec

• Commit: dca4219

Test stats 🧪

Test	Results
Failed	0
Passed	115
Skipped	0
Total	115

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[ChrsMark]

Great addition!

1. Could we also add sample logs along with a pipeline test?
2. Also since I guess there is a specific pattern in these logs, will we be explicit in the fields we are storing here?

[ChrsMark]

Suggested change

	- version: "1.5.1"
	- version: "1.6.0"

New feature ;)

[tetianakravchenko]

@ChrsMark thank you for the feedback!

1. Could we also add sample logs along with a pipeline test?

I will have a look how to add it 👍

2. Also since I guess there is a specific pattern in these logs, will we be explicit in the fields we are storing here?

do you mean specifically kubernetes.audit.*?

[ChrsMark]

2. Also since I guess there is a specific pattern in these logs, will we be explicit in the fields we are storing here?

do you mean specifically kubernetes.audit.*?

Well if the possible fields are unknown then yes we can do something similar to what we do for kubernetes.labels.* or prometheus.labels.*. Otherwise if the fields are specific we can define them like kubernetes.audit.verb etc. Maybe https://kubernetes.io/docs/reference/config-api/apiserver-audit.v1/#audit-k8s-io-v1-Event can be of help here?

[tetianakravchenko]

@ChrsMark should I keep it as experimental ?

[ChrsMark]

I think it should be experimental or beta at least until we have tests+dashboards. @akshay-saraswat could confirm this too.

[tetianakravchenko]

without defined target - decoded json fields will be written under root
related issue - elastic/beats#29422

[tetianakravchenko]

it is not possible to use kubernetes.audit as a target, related issue: elastic/beats#29395

[MichaelKatsoulis]

You can remove declaration of annotation and change the iteration to this

for (var annotation in audit["annotations"]) {

[tetianakravchenko]

done 6dad8d7

[ChrsMark]

Looks good! I left a few thoughts about the fields.

The way that we use the parser and the script processor is the best way possible at the moment I think so we are good here, good job!

Fyi we will need dashboard and tests for this at some point. Maybe a meta issue to keep track of this would help since it's not that much urgent to have them really soon :).

[ChrsMark]

Any reason for not using keyword here? In kubernetes it is defined as keyword: https://github.com/elastic/integrations/blob/master/packages/kubernetes/data_stream/container/fields/base-fields.yml#L57. In general we should use keyword when we want to make this field to be heavily searchable.

Also do you think we could move this field under kubernetes.annotations.* that is already defined for kubernetes? In this way we might be able to corelate events with resources, however I'm not quite sure what values these annotations of the events could take and if related with the annotations of a resource. Let me know what you think.

[tetianakravchenko]

Any reason for not using keyword here? In kubernetes it is defined as keyword: https://github.com/elastic/integrations/blob/master/packages/kubernetes/data_stream/container/fields/base-fields.yml#L57. In general we should use keyword when we want to make this field to be heavily searchable.

I used text, because one of the fields authorization_k8s_io/reason seems to be rather an text according to the name and the example value below:

"annotations": {
      "authorization_k8s_io/decision": "allow",
       "authorization_k8s_io/reason": "RBAC: allowed by ClusterRoleBinding \"system:public-info-viewer\" of ClusterRole \"system:public-info-viewer\" to Group \"system:unauthenticated\""
            },

Do you think it would be better to use keyword instead

Also do you think we could move this field under kubernetes.annotations.* that is already defined for kubernetes? In this way we might be able to corelate events with resources, however I'm not quite sure what values these annotations of the events could take and if related with the annotations of a resource. Let me know what you think.

I think those annotations.* are not related to the resource: from the doc:

Note that these annotations are for the audit event, and do not correspond to the metadata.annotations of the submitted object. Keys should uniquely identify the informing component to avoid name collisions (e.g. podsecuritypolicy.admission.k8s.io/policy). Values should be short. Annotations are included in the Metadata level.

[ChrsMark]

Ok, I think we can leave them as is for now. In any case this is still beta so we will have the chance to iterate on this later if we have feedback.

[ChrsMark]

For shake of simplicity we usually remove hosts's informations from sample events. A very baseline event with the actually interesting fields populated should be enough.

[tetianakravchenko]

thanks for letting me know, I think I've checked few sample_event files in kubernetes package (node, pod) and they actually contain all fields, including host, so the idea is just to keep kubernetes.* here, right?

[ChrsMark]

I would say that yes, the simple the better. I don't see any reason into having all these extra metadata for sample events that want to just show k8s relevant fields. Other data_streams might have been ported automatically and hence this is why they brought sample events from Beats repo that might container all those. I don't think it is documented somewhere however it's rather a "best-practice" convention :).

[tetianakravchenko]

done 74b92dc

[tetianakravchenko]

Fyi we will need dashboard and tests for this at some point. Maybe a meta issue to keep track of this would help since it's not that much urgent to have them really soon :).

@ChrsMark follow up ticket for tests and dashboards: #2343

[ChrsMark]

🚢 it!