Skip to content

Support additional parser configuration: ndjson and multiline in container logs data-stream - #2345

Merged
tetianakravchenko merged 8 commits into
elastic:masterfrom
tetianakravchenko:container-log-json-parser
Dec 17, 2021
Merged

Support additional parser configuration: ndjson and multiline in container logs data-stream#2345
tetianakravchenko merged 8 commits into
elastic:masterfrom
tetianakravchenko:container-log-json-parser

Conversation

@tetianakravchenko

@tetianakravchenko tetianakravchenko commented Dec 15, 2021 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?

  • provide a possibility to configure additional parsers for kubernetes container logs: ndjson and multiline
  • add sample_event and add missing fields
  • add possibility to configure container parser with format and stream configuration parameters

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Related issues

Screenshots

Screenshot 2021-12-16 at 13 57 25

Screenshot 2021-12-16 at 14 01 57

Screenshot 2021-12-16 at 14 21 20

Example of plain txt log:

Example of nginx plain text log 
{
  "_index": ".ds-logs-kubernetes.container_logs-default-2021.12.16-000001",
  "_id": "IWVnw30Bf7XRDSjVGObL",
  "_version": 1,
  "_score": 1,
  "_source": {
    "container": {
      "image": {
        "name": "k8s.gcr.io/kube-proxy:v1.21.1"
      },
      "runtime": "containerd",
      "id": "f2753e543c56ee19d3300d7ac85cc6435414c7be83c053df497ff5bdd659e39f"
    },
    "kubernetes": {
      "container": {
        "name": "kube-proxy"
      },
      "node": {
        "uid": "57ccd748-c877-4be9-9b0e-568e9f205025",
        "hostname": "kind-control-plane",
        "name": "kind-control-plane",
        "labels": {
          "node_kubernetes_io/exclude-from-external-load-balancers": "",
          "node-role_kubernetes_io/master": "",
          "kubernetes_io/hostname": "kind-control-plane",
          "node-role_kubernetes_io/control-plane": "",
          "beta_kubernetes_io/os": "linux",
          "kubernetes_io/arch": "amd64",
          "kubernetes_io/os": "linux",
          "beta_kubernetes_io/arch": "amd64"
        }
      },
      "pod": {
        "uid": "8ff288bc-5508-4232-9c69-5c1bad7bbaca",
        "ip": "172.20.0.2",
        "name": "kube-proxy-gxzks"
      },
      "namespace": "kube-system",
      "namespace_uid": "a4453575-518e-4a21-9909-34874f674177",
      "daemonset": {
        "name": "kube-proxy"
      },
      "namespace_labels": {
        "kubernetes_io/metadata_name": "kube-system"
      },
      "labels": {
        "controller-revision-hash": "6bc6858f58",
        "pod-template-generation": "1",
        "k8s-app": "kube-proxy"
      }
    },
    "agent": {
      "name": "kind-control-plane",
      "id": "f6fca0d7-7b73-4b02-80e9-c46450033048",
      "type": "filebeat",
      "ephemeral_id": "08146946-f5e2-44df-805b-709506fff10c",
      "version": "8.1.0"
    },
    "log": {
      "file": {
        "path": "/var/log/containers/kube-proxy-gxzks_kube-system_kube-proxy-f2753e543c56ee19d3300d7ac85cc6435414c7be83c053df497ff5bdd659e39f.log"
      },
      "offset": 9863
    },
    "elastic_agent": {
      "id": "f6fca0d7-7b73-4b02-80e9-c46450033048",
      "version": "8.1.0",
      "snapshot": true
    },
    "message": "W1216 13:20:46.854871       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice",
    "input": {
      "type": "filestream"
    },
    "orchestrator": {
      "cluster": {
        "name": "kind",
        "url": "kind-control-plane:6443"
      }
    },
    "@timestamp": "2021-12-16T13:20:46.855Z",
    "ecs": {
      "version": "8.0.0"
    },
    "stream": "stderr",
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "kubernetes.container_logs"
    },
    "host": {
      "hostname": "kind-control-plane",
      "os": {
        "kernel": "5.10.47-linuxkit",
        "codename": "Core",
        "name": "CentOS Linux",
        "type": "linux",
        "family": "redhat",
        "version": "7 (Core)",
        "platform": "centos"
      },
      "containerized": true,
      "ip": [
        "10.244.0.1",
        "10.244.0.1",
        "10.244.0.1",
        "172.20.0.2",
        "172.18.0.2",
        "fc00:f853:ccd:e793::2",
        "fe80::42:acff:fe12:2"
      ],
      "name": "kind-control-plane",
      "id": "85e35c2b5e1b39ba72393a6baf6ee7cd",
      "mac": [
        "4a:71:e6:e3:51:8a",
        "76:9b:c8:36:6f:9c",
        "42:c4:d2:ad:e0:3d",
        "02:42:ac:14:00:02",
        "02:42:ac:12:00:02"
      ],
      "architecture": "x86_64"
    },
    "event": {
      "agent_id_status": "verified",
      "ingested": "2021-12-16T13:20:53Z",
      "dataset": "kubernetes.container_logs"
    }
  },
  "fields": {
    "kubernetes.node.uid": [
      "57ccd748-c877-4be9-9b0e-568e9f205025"
    ],
    "orchestrator.cluster.name": [
      "kind"
    ],
    "kubernetes.daemonset.name": [
      "kube-proxy"
    ],
    "elastic_agent.version": [
      "8.1.0"
    ],
    "kubernetes.namespace_uid": [
      "a4453575-518e-4a21-9909-34874f674177"
    ],
    "host.os.name.text": [
      "CentOS Linux"
    ],
    "host.hostname": [
      "kind-control-plane"
    ],
    "kubernetes.node.labels.kubernetes_io/os": [
      "linux"
    ],
    "host.mac": [
      "4a:71:e6:e3:51:8a",
      "76:9b:c8:36:6f:9c",
      "42:c4:d2:ad:e0:3d",
      "02:42:ac:14:00:02",
      "02:42:ac:12:00:02"
    ],
    "container.id": [
      "f2753e543c56ee19d3300d7ac85cc6435414c7be83c053df497ff5bdd659e39f"
    ],
    "kubernetes.node.labels.node-role_kubernetes_io/master": [
      ""
    ],
    "container.image.name": [
      "k8s.gcr.io/kube-proxy:v1.21.1"
    ],
    "host.os.version": [
      "7 (Core)"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/os": [
      "linux"
    ],
    "kubernetes.namespace": [
      "kube-system"
    ],
    "host.os.name": [
      "CentOS Linux"
    ],
    "agent.name": [
      "kind-control-plane"
    ],
    "host.name": [
      "kind-control-plane"
    ],
    "kubernetes.labels.k8s-app": [
      "kube-proxy"
    ],
    "event.agent_id_status": [
      "verified"
    ],
    "host.os.type": [
      "linux"
    ],
    "input.type": [
      "filestream"
    ],
    "log.offset": [
      9863
    ],
    "data_stream.type": [
      "logs"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "container.runtime": [
      "containerd"
    ],
    "agent.id": [
      "f6fca0d7-7b73-4b02-80e9-c46450033048"
    ],
    "host.containerized": [
      true
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "kubernetes.node.labels.node-role_kubernetes_io/control-plane": [
      ""
    ],
    "agent.version": [
      "8.1.0"
    ],
    "host.os.family": [
      "redhat"
    ],
    "kubernetes.node.name": [
      "kind-control-plane"
    ],
    "kubernetes.node.hostname": [
      "kind-control-plane"
    ],
    "kubernetes.pod.uid": [
      "8ff288bc-5508-4232-9c69-5c1bad7bbaca"
    ],
    "host.ip": [
      "10.244.0.1",
      "10.244.0.1",
      "10.244.0.1",
      "172.20.0.2",
      "172.18.0.2",
      "fc00:f853:ccd:e793::2",
      "fe80::42:acff:fe12:2"
    ],
    "agent.type": [
      "filebeat"
    ],
    "orchestrator.cluster.url": [
      "kind-control-plane:6443"
    ],
    "stream": [
      "stderr"
    ],
    "host.os.kernel": [
      "5.10.47-linuxkit"
    ],
    "kubernetes.pod.name": [
      "kube-proxy-gxzks"
    ],
    "elastic_agent.snapshot": [
      true
    ],
    "host.id": [
      "85e35c2b5e1b39ba72393a6baf6ee7cd"
    ],
    "kubernetes.pod.ip": [
      "172.20.0.2"
    ],
    "kubernetes.container.name": [
      "kube-proxy"
    ],
    "elastic_agent.id": [
      "f6fca0d7-7b73-4b02-80e9-c46450033048"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "host.os.codename": [
      "Core"
    ],
    "kubernetes.namespace_labels.kubernetes_io/metadata_name": [
      "kube-system"
    ],
    "kubernetes.labels.pod-template-generation": [
      "1"
    ],
    "message": [
      "W1216 13:20:46.854871       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice"
    ],
    "kubernetes.node.labels.kubernetes_io/hostname": [
      "kind-control-plane"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/arch": [
      "amd64"
    ],
    "event.ingested": [
      "2021-12-16T13:20:53.000Z"
    ],
    "@timestamp": [
      "2021-12-16T13:20:46.855Z"
    ],
    "host.os.platform": [
      "centos"
    ],
    "kubernetes.labels.controller-revision-hash": [
      "6bc6858f58"
    ],
    "log.file.path": [
      "/var/log/containers/kube-proxy-gxzks_kube-system_kube-proxy-f2753e543c56ee19d3300d7ac85cc6435414c7be83c053df497ff5bdd659e39f.log"
    ],
    "data_stream.dataset": [
      "kubernetes.container_logs"
    ],
    "agent.ephemeral_id": [
      "08146946-f5e2-44df-805b-709506fff10c"
    ],
    "kubernetes.node.labels.kubernetes_io/arch": [
      "amd64"
    ],
    "kubernetes.node.labels.node_kubernetes_io/exclude-from-external-load-balancers": [
      ""
    ],
    "event.dataset": [
      "kubernetes.container_logs"
    ]
  }
}

Example of json logs

Example of elastic-agent json text log 
{
  "_index": ".ds-logs-kubernetes.container_logs-default-2021.12.16-000001",
  "_id": "1GVmw30Bf7XRDSjV_eVD",
  "_version": 1,
  "_score": 1,
  "_source": {
    "container": {
      "image": {
        "name": "docker.elastic.co/beats/elastic-agent:8.1.0-SNAPSHOT"
      },
      "runtime": "containerd",
      "id": "47eef6ed0b3de13ac150ba7adf4e5cb2cc8d06efb13ce7ee58b7e851c2ab0f21"
    },
    "kubernetes": {
      "container": {
        "name": "elastic-agent"
      },
      "node": {
        "uid": "57ccd748-c877-4be9-9b0e-568e9f205025",
        "hostname": "kind-control-plane",
        "name": "kind-control-plane",
        "labels": {
          "node_kubernetes_io/exclude-from-external-load-balancers": "",
          "node-role_kubernetes_io/master": "",
          "kubernetes_io/hostname": "kind-control-plane",
          "node-role_kubernetes_io/control-plane": "",
          "beta_kubernetes_io/os": "linux",
          "kubernetes_io/arch": "amd64",
          "kubernetes_io/os": "linux",
          "beta_kubernetes_io/arch": "amd64"
        }
      },
      "pod": {
        "uid": "370d764e-f1b9-4932-934f-b415bae76b7d",
        "ip": "172.20.0.2",
        "name": "elastic-agent-tst5x"
      },
      "namespace": "kube-system",
      "daemonset": {
        "name": "elastic-agent"
      },
      "namespace_uid": "a4453575-518e-4a21-9909-34874f674177",
      "namespace_labels": {
        "kubernetes_io/metadata_name": "kube-system"
      },
      "labels": {
        "app": "elastic-agent",
        "controller-revision-hash": "77bd57578",
        "pod-template-generation": "3"
      }
    },
    "agent": {
      "name": "kind-control-plane",
      "id": "f6fca0d7-7b73-4b02-80e9-c46450033048",
      "ephemeral_id": "08146946-f5e2-44df-805b-709506fff10c",
      "type": "filebeat",
      "version": "8.1.0"
    },
    "log": {
      "file": {
        "path": "/var/log/containers/elastic-agent-tst5x_kube-system_elastic-agent-47eef6ed0b3de13ac150ba7adf4e5cb2cc8d06efb13ce7ee58b7e851c2ab0f21.log"
      },
      "offset": 105583
    },
    "elastic_agent": {
      "id": "f6fca0d7-7b73-4b02-80e9-c46450033048",
      "version": "8.1.0",
      "snapshot": true
    },
    "input": {
      "type": "filestream"
    },
    "orchestrator": {
      "cluster": {
        "name": "kind",
        "url": "kind-control-plane:6443"
      }
    },
    "@timestamp": "2021-12-16T13:20:41.251Z",
    "ecs": {
      "version": "8.0.0"
    },
    "stream": "stderr",
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "kubernetes.container_logs"
    },
    "host": {
      "hostname": "kind-control-plane",
      "os": {
        "kernel": "5.10.47-linuxkit",
        "codename": "Core",
        "name": "CentOS Linux",
        "type": "linux",
        "family": "redhat",
        "version": "7 (Core)",
        "platform": "centos"
      },
      "containerized": true,
      "ip": [
        "10.244.0.1",
        "10.244.0.1",
        "10.244.0.1",
        "172.20.0.2",
        "172.18.0.2",
        "fc00:f853:ccd:e793::2",
        "fe80::42:acff:fe12:2"
      ],
      "name": "kind-control-plane",
      "id": "85e35c2b5e1b39ba72393a6baf6ee7cd",
      "mac": [
        "4a:71:e6:e3:51:8a",
        "76:9b:c8:36:6f:9c",
        "42:c4:d2:ad:e0:3d",
        "02:42:ac:14:00:02",
        "02:42:ac:12:00:02"
      ],
      "architecture": "x86_64"
    },
    "json": {
      "log.origin": {
        "file.line": 66,
        "file.name": "stateresolver/stateresolver.go"
      },
      "@timestamp": "2021-12-16T13:20:41.251Z",
      "ecs.version": "1.6.0",
      "log.level": "info",
      "message": "Updating internal state"
    },
    "event": {
      "agent_id_status": "verified",
      "ingested": "2021-12-16T13:20:46Z",
      "dataset": "kubernetes.container_logs"
    }
  },
  "fields": {
    "kubernetes.node.uid": [
      "57ccd748-c877-4be9-9b0e-568e9f205025"
    ],
    "orchestrator.cluster.name": [
      "kind"
    ],
    "kubernetes.daemonset.name": [
      "elastic-agent"
    ],
    "elastic_agent.version": [
      "8.1.0"
    ],
    "kubernetes.namespace_uid": [
      "a4453575-518e-4a21-9909-34874f674177"
    ],
    "host.os.name.text": [
      "CentOS Linux"
    ],
    "host.hostname": [
      "kind-control-plane"
    ],
    "kubernetes.node.labels.kubernetes_io/os": [
      "linux"
    ],
    "host.mac": [
      "4a:71:e6:e3:51:8a",
      "76:9b:c8:36:6f:9c",
      "42:c4:d2:ad:e0:3d",
      "02:42:ac:14:00:02",
      "02:42:ac:12:00:02"
    ],
    "container.id": [
      "47eef6ed0b3de13ac150ba7adf4e5cb2cc8d06efb13ce7ee58b7e851c2ab0f21"
    ],
    "kubernetes.node.labels.node-role_kubernetes_io/master": [
      ""
    ],
    "container.image.name": [
      "docker.elastic.co/beats/elastic-agent:8.1.0-SNAPSHOT"
    ],
    "kubernetes.labels.app": [
      "elastic-agent"
    ],
    "host.os.version": [
      "7 (Core)"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/os": [
      "linux"
    ],
    "kubernetes.namespace": [
      "kube-system"
    ],
    "host.os.name": [
      "CentOS Linux"
    ],
    "agent.name": [
      "kind-control-plane"
    ],
    "host.name": [
      "kind-control-plane"
    ],
    "event.agent_id_status": [
      "verified"
    ],
    "host.os.type": [
      "linux"
    ],
    "json.log.level": [
      "info"
    ],
    "input.type": [
      "filestream"
    ],
    "log.offset": [
      105583
    ],
    "data_stream.type": [
      "logs"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "container.runtime": [
      "containerd"
    ],
    "agent.id": [
      "f6fca0d7-7b73-4b02-80e9-c46450033048"
    ],
    "host.containerized": [
      true
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "kubernetes.node.labels.node-role_kubernetes_io/control-plane": [
      ""
    ],
    "agent.version": [
      "8.1.0"
    ],
    "host.os.family": [
      "redhat"
    ],
    "kubernetes.node.name": [
      "kind-control-plane"
    ],
    "json.@timestamp": [
      "2021-12-16T13:20:41.251Z"
    ],
    "kubernetes.node.hostname": [
      "kind-control-plane"
    ],
    "kubernetes.pod.uid": [
      "370d764e-f1b9-4932-934f-b415bae76b7d"
    ],
    "host.ip": [
      "10.244.0.1",
      "10.244.0.1",
      "10.244.0.1",
      "172.20.0.2",
      "172.18.0.2",
      "fc00:f853:ccd:e793::2",
      "fe80::42:acff:fe12:2"
    ],
    "agent.type": [
      "filebeat"
    ],
    "orchestrator.cluster.url": [
      "kind-control-plane:6443"
    ],
    "stream": [
      "stderr"
    ],
    "host.os.kernel": [
      "5.10.47-linuxkit"
    ],
    "kubernetes.pod.name": [
      "elastic-agent-tst5x"
    ],
    "json.ecs.version": [
      "1.6.0"
    ],
    "elastic_agent.snapshot": [
      true
    ],
    "json.log.origin.file.line": [
      66
    ],
    "host.id": [
      "85e35c2b5e1b39ba72393a6baf6ee7cd"
    ],
    "kubernetes.pod.ip": [
      "172.20.0.2"
    ],
    "kubernetes.container.name": [
      "elastic-agent"
    ],
    "elastic_agent.id": [
      "f6fca0d7-7b73-4b02-80e9-c46450033048"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "host.os.codename": [
      "Core"
    ],
    "json.message": [
      "Updating internal state"
    ],
    "kubernetes.namespace_labels.kubernetes_io/metadata_name": [
      "kube-system"
    ],
    "kubernetes.labels.pod-template-generation": [
      "3"
    ],
    "kubernetes.node.labels.kubernetes_io/hostname": [
      "kind-control-plane"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/arch": [
      "amd64"
    ],
    "event.ingested": [
      "2021-12-16T13:20:46.000Z"
    ],
    "@timestamp": [
      "2021-12-16T13:20:41.251Z"
    ],
    "host.os.platform": [
      "centos"
    ],
    "kubernetes.labels.controller-revision-hash": [
      "77bd57578"
    ],
    "json.log.origin.file.name": [
      "stateresolver/stateresolver.go"
    ],
    "log.file.path": [
      "/var/log/containers/elastic-agent-tst5x_kube-system_elastic-agent-47eef6ed0b3de13ac150ba7adf4e5cb2cc8d06efb13ce7ee58b7e851c2ab0f21.log"
    ],
    "data_stream.dataset": [
      "kubernetes.container_logs"
    ],
    "agent.ephemeral_id": [
      "08146946-f5e2-44df-805b-709506fff10c"
    ],
    "kubernetes.node.labels.kubernetes_io/arch": [
      "amd64"
    ],
    "kubernetes.node.labels.node_kubernetes_io/exclude-from-external-load-balancers": [
      ""
    ],
    "event.dataset": [
      "kubernetes.container_logs"
    ]
  }
}
@tetianakravchenko
add sample_event; support ndjson parser
6a420a6 
Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko
add possibility to adjust container, ndjson and multiline parser conf…
f6c841d 
…iguratios

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko
add pr link
5ea34dc 
Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@elasticmachine

elasticmachine commented Dec 15, 2021 •
edited
Loading

Copy link
Copy Markdown

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-12-17T11:31:24.736+0000

  • Duration: 30 min 27 sec

  • Commit: 37be361

Test stats 🧪

Test Results
Failed 0
Passed 116
Skipped 0
Total 116

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.
@tetianakravchenko
remove test; add container parser configuration; use generic Addition…
9f828c4 
…al parsers configuration instead of multilineParser ad jsonParser

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko

tetianakravchenko commented Dec 16, 2021

Copy link
Copy Markdown
Contributor Author

@mukeshelastic @akshay-saraswat please have a look on the screenshots in the PR description, and let me know if you have any objections on the UI for this feature
@tetianakravchenko tetianakravchenko added the enhancement New feature or request label Dec 16, 2021
@tetianakravchenko tetianakravchenko changed the title Support parsing json logs with kubernetes integration Dec 16, 2021
@tetianakravchenko
add a link to parsers documentation
783b598 
Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko

tetianakravchenko commented Dec 17, 2021

Copy link
Copy Markdown
Contributor Author

we also discussed with @mukeshelastic option to enable json parser by default, for now was decided to keep it as a advanced configuration, as this configuration applies to all cluster and there is no conditions available to limit it to specified image/namespace. It might be changed later
@tetianakravchenko

tetianakravchenko commented Dec 17, 2021

Copy link
Copy Markdown
Contributor Author

/test
ChrsMark
ChrsMark reviewed Dec 17, 2021
View reviewed changes

@ChrsMark ChrsMark left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I left a few comments.
Comment thread packages/kubernetes/changelog.yml Outdated
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.7.1"

@ChrsMark ChrsMark Dec 17, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an enhancement so I would go with 1.8.0

@tetianakravchenko tetianakravchenko Dec 17, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done a156f50
Comment thread packages/kubernetes/data_stream/container_logs/manifest.yml Outdated
default: |
# - ndjson:
# target: json
# - multiline: ~

@ChrsMark ChrsMark Dec 17, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It this is the baseline config for multiline or just a placeholder? Maybe a more accurate default could be used here.

@tetianakravchenko tetianakravchenko Dec 17, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would consider it more as a placeholder, to be honest I am not sure what could be a reasonable default here, any ideas here?

@ChrsMark ChrsMark Dec 17, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe sth like

multiline.type: pattern
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after

The very basic example from https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html#multiline. Of course this is not supposed to cover 100% the possible cases so it would be nice if we can refer to docs so as to be be more clear. The idea of having a more complete config here is to give a sense of how the config looks like and then users can dive more into the docs according their specific use cases.

@tetianakravchenko tetianakravchenko Dec 17, 2021 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ChrsMark I think it makes sense to add it, at least to show how this will look like as a parser. I've added multiline config - 37be361
also run a quick test: after enabling

- multiline:
    type: pattern
    pattern: '^\['
    negate: true
    match: after

it seems to work as expected (examle - kibana logs)
Screenshot 2021-12-17 at 12 30 26
Comment thread packages/kubernetes/data_stream/container_logs/fields/base-fields.yml
description: >
Kubernetes hostname as reported by the node’s kernel

- name: node.labels.*

@ChrsMark ChrsMark Dec 17, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we will also need to document node.annotations.* even if they are not generated by default and hence the tests would never warn us about having them undocumented. Same for namespace_annotations.

@tetianakravchenko tetianakravchenko Dec 17, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done a156f50
@tetianakravchenko
bump package version to 1.8.0; add node.annotations. and namespace_an…
a156f50 
…notations.* fields

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko
add container parser link to the documentation
f0a23bd 
Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
ChrsMark
ChrsMark approved these changes Dec 17, 2021
View reviewed changes

@ChrsMark ChrsMark left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, good job here!

I left a small suggestion about the multiline, feel free to add it or reject it. Otherwise if CI is happy feel free to merge it!
@MichaelKatsoulis

MichaelKatsoulis commented Dec 17, 2021

Copy link
Copy Markdown
Contributor

Great job here @tetianakravchenko !
@tetianakravchenko
add multiline configuration example
37be361 
Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko
tetianakravchenko merged commit 1dbbe4e into elastic:master Dec 17, 2021
@tetianakravchenko tetianakravchenko mentioned this pull request Dec 23, 2021
Merged
6 tasks
orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
@tetianakravchenko
Support additional parser configuration: ndjson and multiline in cont…
ca3ee3b 
…ainer logs data-stream (elastic#2345)

* add sample_event; support ndjson parser

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* add possibility to adjust container, ndjson and multiline parser configuratios

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* add pr link

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* remove test; add container parser configuration; use generic Additional parsers configuration instead of multilineParser ad jsonParser

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* add a link to parsers documentation

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* bump package version to 1.8.0; add node.annotations. and namespace_annotations.* fields

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* add container parser link to the documentation

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* add multiline configuration example

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

4 participants

@tetianakravchenko @elasticmachine @MichaelKatsoulis @ChrsMark