[kubernetes#container_logs] allow to edit ID and add docs on how to ingest rotated logs

[AndersonQ]

Proposed commit message

    [kubernetes] update container logs
    
     - Allow to configure input ID.
     - Update container logs docs to include how to ingest rotated log files, including GZIP-compressed files.

Checklist

• [ ] I have reviewed tips for building integrations and this pull request is aligned with them.
• [ ] I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• [ ] I have verified that Kibana version constraints are current according to guidelines.
• [ ] I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Create an elastic stack
• setup a kind cluster

kind create cluster

• deploy the agent to the cluster following the instructions in kibana. If you see the error InvalidImageName, check here to see hot to fix it
• add the k8s integration from the fleet UI
• start some flog containers:

kubectl apply -f ./flog.yaml

flog.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: flog-log-generator
spec:
  template:
    spec:
      containers:
        - name: flog-unstructured-cont-rot
          image: mingrammer/flog
          #          too small "-d" won't give kubelet time to rotate the files
          args: ["-t", "stdout", "-d", "1us", "-l"]
        - name: crashloop-1
          image: busybox
          imagePullPolicy: IfNotPresent
          command: ["sh", "-c", "echo 'Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vel illum dolore eu feugiat nulla facilisis at' >&2; sleep 1; exit 1"]
      restartPolicy: OnFailure
  backoffLimit: 10

• ensure the logs are being ingested. Use filter log.file.path : *flog*
• build, then install the integration

cd packages/kubernetes
elastic-package build -v
elastic-package install -v

• upgrade the integration
• check all is normal, no configuration has changed
• follow the docs to ingest the rotated logs
• check:
  • there is a spike in ingested logs:

• check the crashloop-1 container logs to see it reingested the logs using the new path

• check the rotated logs were ingested. Use the following filters log.file.path : *flog*log.* and log.file.path : *flog*log.*.gz:

• get an agent diagnostic and inspect filestream configuration components/filestream-default/beat-rendered-config.yml to ensure the ID has the correct pattern and the path is correct:

      id: kubernetes-container-logs-83c0d508-bd6b-45c5-88f7-9f08a56a3563-kube-controller-manager
      index: logs-kubernetes.container_logs-default
      parsers:
        - container:
            format: auto
            stream: all
      paths:
        - /var/log/pods/kube-system_kube-controller-manager-kind-control-plane_83c0d508-bd6b-45c5-88f7-9f08a56a3563/kube-controller-manager/*.log*

Disruptive User Impact

When enabling the ingestion of rotated logs on an existing deployment, it cause a one-time re-ingestion of the logs.

The input ID when using autodiscover is now scoped by container name, instead of container id. It's done to avoid data duplication as explained below.

Context: K8s logging architecture

Logs are written to:

/var/log/pods/<namespace>_<pod_name>_<pod_uid>/<container_name>/<restart_count>.log # Active log file
/var/log/pods/<namespace>_<pod_name>_<pod_uid>/<container_name>/<restart_count>.log.TIMRESTAMP` # 1st rotation, plain file
/var/log/pods/<namespace>_<pod_name>_<pod_uid>/<container_name>/<restart_count>.log.TIMRESTAMP.gz` # subsequenmtt rotations, gziped

Legacy symlinked path:

/var/log/containers/<pod_name>_<namespace>_<container_name>-<container_id>.log -> /var/log/pods/<namespace>_<pod_name>_<pod_uid>/<container_name>/<restart_count>.log

Thus during a container restart, a CrashLoopBackoff, it's possible to have 2
containers with the same name, but different IDs. The container which just crashed
and is being removed and the new container being created.

Just for the example, lets consider:

container name: foo
container id 1: id-1
container id 2: id-2

pod name: pod
namespace: ns
pod uid: uid

This leads to the following log files and symlinks during the transition:

/var/log/pods/ns_pod_uid/foo/0.log
/var/log/pods/ns_pod_uid/foo/1.log

/var/log/containers/pod_ns_foo-id-1.log -> /var/log/pods/ns_pod_uid/foo/0.log
/var/log/containers/pod_ns_foo-id-2.log -> /var/log/pods/ns_pod_uid/foo/1.log

k8s log collection

1 filestream input per container. Input ID pattern: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}

Thus, for each container id, one input is created.

Rotated logs

Rotated logs are scoped by container name, and restart counter, not container ID. (/var/log/pods/<pod_name><pod_uid>/<container_name>/<restart_count>.log)
To collect rotated logs, the following wildcard would be used:

/var/log/pods/<namespace>_<pod_name>_<pod_uid>/<container_name>/*.log.*

id: kubernetes-container-logs-${kubernetes.pod.name}-${kubernetes.container.id}
paths:
  - /var/log/containers/*<container_id>.log
  - /var/log/pods/<namespace>_<pod_name>_<pod_uid>/<container_name>/*.log.*

for the example above, during the container restart, there would be 2 inputs created:

id: kubernetes-container-logs-pod-id-1
paths:
  - /var/log/containers/*id-1.log
  - /var/log/pods/ns_pod_uid/foo/*.log.*

id: kubernetes-container-logs-pod-id-2
paths:
  - /var/log/containers/*id-2.log
  - /var/log/pods/ns_pod_uid/foo/*.log.*

Both inputs harvest /var/log/pods/ns_pod_uid/foo/*.log.*, duplicating the data.

Therefore, the ID pattern must be scoped by container name.

Related issues

• Closes [kubernetes] update docs to explain how to ingest rotated log files (including GZIP) #15792

Screenshots

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elasticmachine]

Pinging @elastic/ecosystem (Team:Ecosystem)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[faec]

One small nit, otherwise looks good

[gizas]

Suggested change

	- change the _ID_ to `kubernetes-container-logs-${kubernetes.pod.uid}-${kubernetes.container.name}`
	- change the `ID` to `kubernetes-container-logs-${kubernetes.pod.uid}-${kubernetes.container.name}`

[gizas]

You are refering to the ID field right? ID the unique identifier for the input. So the ID can b misleading

[AndersonQ]

You mean if it's ID instead of ID? Or that ID isn't clear enough?

[gizas]

nit: maybe the _ ID_ can be misleading. minor

[gizas]

Maybe you can add here in the description the new ID: /var/log/pods/${kubernetes.namespace}_${kubernetes.pod.name}_${kubernetes.pod.uid}/${kubernetes.container.name}/*.log* that should be added here in case gzip_experimental: true or a link to this doc

[AndersonQ]

done in fb0869f

[jsoriano]

Do we expect more possible values for this? Wondering if this could be a boolean option, that changes the defaults.

I guess we are not making this the default to avoid breaking changes? Or this is not available in all clusters?

[AndersonQ]

the problem is we do not have to know if there is a custom configuration, if the user changed the defaults. The toggle would override everything, regardless if it's the default or not. That's why I did not use a toggle

[AndersonQ]

@gizas, I made the changes you requested

@jsoriano I made the changes you requested, if you wanna review again, please do so :)

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: fb0869f

History

• 💚 Build #34308 succeeded 1639cff
• 💚 Build #34132 succeeded 41cb1f3
• 💚 Build #33957 succeeded bf7e169
• 💚 Build #33724 succeeded 3464497
• 💚 Build #33339 succeeded 61774cb

cc @AndersonQ

[elastic-vault-github-plugin-prod]

Package kubernetes - 1.82.0 containing this change is available at https://epr.elastic.co/package/kubernetes/1.82.0/