Migrate kubernetes module by ChrsMark · Pull Request #70 · elastic/integrations

ChrsMark · 2020-06-11T09:18:40Z

Implementing Kubernetes package migration
Important Notes:

container, system, node, pod, volume metricsets require access to the Kubelet API. Usually these are enabled when Metricbeat is deployed as Daemonset on k8s nodes.
state_* metricsets require the deployment kube-state-metrics on the k8s cluster in order to retrieve metrics from it.
apiserver, scheduler, proxy and controllermanager require access to each component of the k8s cluster accordingly.

Wondering how all these would be possible in combination with agent. Do we need to define how agent will run in k8s (deployment, daemonset) in order to make it work? @exekias do you have anything in mind about this?

Docs: https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-kubernetes.html

Migration Steps:

Screenshots

Manual Testing

Using the testing env introduced at Add k8s manifests to deploy stack along with Agent #89 we spin up the Stack and Agent on a k8s cluster.
For the 2 different configs (see Add k8s manifests to deploy stack along with Agent #89) enable the Package once for node scope Agent and once for cluster scope.
Verify that all datasets are shipping metrics and Dashboards are being populated.

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

elasticmachine · 2020-06-11T09:21:37Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Branch indexing]
Start Time: 2020-06-29T17:25:26.637+0000
Duration: 4 min 49 sec

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark · 2020-06-24T12:28:27Z

@exekias @mtojek this one seems to be completed in terms of "package migration". Would love your feedback here.

Manual testing is also possible already (using https://github.com/elastic/integrations-dev/issues/58) in order to verify that data are being collected and Dashboards work ok.

We still need to verify what will happen with Agent's side regarding docs and manifests around the Deployment strategies on k8s.

mtojek · 2020-06-24T14:35:00Z

I think you can convert it to ordinary pull-request. Thanks for the contribution.

exekias

Didn't test it, but it looks good. I think we need to improve docs, left a comment on that regard

exekias · 2020-06-25T09:01:33Z

@@ -0,0 +1,38 @@
+title: Kubernetes apiserver metrics


Suggested change

title: Kubernetes apiserver metrics

title: Kubernetes API Server metrics

exekias · 2020-06-25T09:03:40Z

+  title: Kubernetes apiserver metrics
+  description: Collect Kubernetes apiserver metrics


Suggested change

title: Kubernetes apiserver metrics

description: Collect Kubernetes apiserver metrics

title: Kubernetes API Server metrics

description: Collect Kubernetes API Server metrics

exekias · 2020-06-25T09:27:42Z

+
+This integration is used to collect metrics from 
+[Kubernetes clusters](https://kubernetes.io/).
+


I'm missing many of the explanations that we have here: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html

It would be nice to provide good understanding on the variety of datasets that we have and how they need to be configured

Nice catch! Added!

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark · 2020-06-25T12:51:02Z

Updated the Docs and changed the sample json events with new ones according to the new indexing approach.

mtojek · 2020-06-26T06:25:42Z

@@ -0,0 +1,2 @@
+- name: kubernetes
+  type: group


I think that according to latest changes in import-beat, pushed by @kaiyan-sheng , this empty file is not required.

mtojek · 2020-06-26T06:25:55Z

@@ -0,0 +1,2 @@
+- name: kubernetes


mtojek · 2020-06-26T06:26:48Z

@@ -0,0 +1 @@
+metricsets: ["event"]


Does it need any configuration options? There is even no "period" defined hence wondering.

I added the configuration manually in the manifest.yml and missed that. 👍

mtojek · 2020-06-26T06:33:54Z

@@ -0,0 +1,44 @@
+title: Kubernetes Node metrics
+release: experimental


Please adjusts all release fields. We will release all new packages as "beta".

mtojek · 2020-06-26T06:36:02Z

+    show_user: true
+    default:
+    - kube-state-metrics:8080
+  title: Kubernetes state_resourcequota metrics


I think you can update the title to a pleasant, human-readable form (no underscore).

mtojek · 2020-06-26T06:37:00Z

+format_version: 1.0.0
+name: kubernetes
+title: Kubernetes
+version: 0.0.1


version: 0.1.0

mtojek · 2020-06-26T06:37:34Z

+requirement:
+  kibana:
+    versions: '>=7.3.0 <8.0.0'
+  elasticsearch: {}


This field (elasticsearch) can be removed.

mtojek · 2020-06-26T06:37:44Z

+categories:
+- metrics
+release: beta
+removable: true


This field too.

mtojek · 2020-06-26T06:38:20Z

+  description: Collect metrics from Kubernetes
+  inputs:
+  - type: kubernetes/metrics
+    title: Collect Kubernetes API Server, Container, Controller Manager, Event, Node, Pod, Proxy, Scheduler, state_container, state_cronjob, state_deployment, state_node, state_persistentvolume, state_persistentvolumeclaim, state_pod, state_replicaset, state_resourcequota, state_service, state_statefulset, state_storageclass, System and Volume metrics


I believe you can shorten this one :D

simply - Collect Kubernetes metrics?

mtojek · 2020-06-26T06:39:05Z

+  inputs:
+  - type: kubernetes/metrics
+    title: Collect Kubernetes API Server, Container, Controller Manager, Event, Node, Pod, Proxy, Scheduler, state_container, state_cronjob, state_deployment, state_node, state_persistentvolume, state_persistentvolumeclaim, state_pod, state_replicaset, state_resourcequota, state_service, state_statefulset, state_storageclass, System and Volume metrics
+    description: Collecting API Server, Container, Controller Manager, Event, Node, Pod, Proxy, Scheduler, state_container, state_cronjob, state_deployment, state_node, state_persistentvolume, state_persistentvolumeclaim, state_pod, state_replicaset, state_resourcequota, state_service, state_statefulset, state_storageclass, System and Volume metrics from Kubernetes


I think that all captions, descriptions, etc. should present names which are human readable (no underscores).

mtojek · 2020-06-26T06:50:26Z

Just noticed on screenshots, please improve case sensitivity on screenshots.

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark · 2020-06-26T09:13:07Z

Thanks for the review @mtojek! Comments should be addressed now.

mtojek

Good job, Christos!

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark · 2020-06-26T11:51:49Z

@exekias @andresrc this integration is completed. Tested manually using testing/environments/kubernetes. Let me know if we can merge/ship it already or we want to wait for a Running Agent on k8s part on Agent's side.

mtojek · 2020-06-30T06:54:36Z

@ChrsMark

I think you have to merge this PR, otherwise you'll have to rebase against the master branch, as there are some breaking changes coming: #132

exekias · 2020-06-30T07:37:06Z

++ Let's get this in. I think it would be nice to follow up with docs on the Agent side on how to deploy in Kubernetes

Merging this on behalf of @ChrsMark, as he is off

Add automated files

d5e653d

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Add README placeholder

a7c4120

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

mtojek marked this pull request as draft June 12, 2020 06:51

ChrsMark added 2 commits June 17, 2020 11:24

release beta

a61eea0

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Add k8s manifests to deploy stack along with Agent

5c19e4c

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

exekias added the Team:Platforms label Jun 18, 2020

ChrsMark mentioned this pull request Jun 18, 2020

Add k8s manifests to deploy stack along with Agent #89

Merged

ChrsMark added 6 commits June 24, 2020 12:14

Fix manifests

ca87d38

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Unify hosts & add overview screenshot

e091568

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Fix icon name

418132a

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Add ReadMe.md

be5c10c

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Add more content in DEADME.md

de8f415

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

final README.md

bfd179a

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark marked this pull request as ready for review June 25, 2020 07:22

mtojek changed the title ~~[WIP] Migrate kubernetes module~~ Jun 25, 2020

exekias reviewed Jun 25, 2020

View reviewed changes

ChrsMark added 3 commits June 25, 2020 14:24

Improve docs

eb74646

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Add more docs

7e95f8e

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Update data.jon in READMEs

6491456

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

exekias approved these changes Jun 25, 2020

View reviewed changes

mtojek reviewed Jun 26, 2020

View reviewed changes

ChrsMark added 4 commits June 26, 2020 10:42

Merge remote-tracking branch 'upstream/master' into migrate_k8s

be1e57b

Merge remote-tracking branch 'upstream/master' into migrate_k8s

1275bcf

Remove empty package-fields files

ed1d121

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

fix

f23f659

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark added 4 commits June 26, 2020 11:50

Fix release fields

ed071ab

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Fix version

2b2a72e

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

More review changes

b945c96

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Fix screenshots

83b9d46

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark requested a review from mtojek June 26, 2020 09:13

mtojek approved these changes Jun 26, 2020

View reviewed changes

ChrsMark added 2 commits June 26, 2020 12:41

Remove leftover k8s manifests

217990a

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Minor descritpion fix

71a97d8

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

exekias merged commit eca38ee into elastic:master Jun 30, 2020

ChrsMark mentioned this pull request Jul 8, 2020

Elastic agent on k8s elastic/beats#19727

Merged

This was referenced Jul 28, 2020

Cherry-pick #19727 to 7.9: Elastic agent on k8s elastic/beats#20262

Merged

Cherry-pick #19727 to 7.x: Elastic agent on k8s elastic/beats#20263

Merged

tetianakravchenko mentioned this pull request Nov 15, 2021

Fix visualizations on the Kubernetes overview dashboard #2151

Merged

4 tasks

andrewkroh added Integration:kubernetes Kubernetes New Integration Issue or pull request for creating a new integration package. labels Aug 13, 2024

JohannesMahne mentioned this pull request Jan 13, 2026

[Chargeback] Fix chargeback visualizations and add automated config lookup #16936

Merged

Navigation Menu

Uh oh!

Migrate kubernetes module - #70

Migrate kubernetes module#70
exekias merged 23 commits into
elastic:masterfrom
ChrsMark:migrate_k8s

ChrsMark commented Jun 11, 2020 •

edited

Loading

elasticmachine commented Jun 11, 2020 •

edited

Loading

Build stats

ChrsMark commented Jun 24, 2020

mtojek commented Jun 24, 2020

exekias left a comment

exekias Jun 25, 2020

exekias Jun 25, 2020

exekias Jun 25, 2020

ChrsMark Jun 25, 2020

ChrsMark commented Jun 25, 2020

mtojek Jun 26, 2020

mtojek Jun 26, 2020

mtojek Jun 26, 2020

ChrsMark Jun 26, 2020

mtojek Jun 26, 2020

mtojek Jun 26, 2020

mtojek Jun 26, 2020

mtojek Jun 26, 2020

mtojek Jun 26, 2020

mtojek Jun 26, 2020

mtojek Jun 26, 2020

mtojek commented Jun 26, 2020 •

edited

Loading

ChrsMark commented Jun 26, 2020

mtojek left a comment

ChrsMark commented Jun 26, 2020 •

edited

Loading

mtojek commented Jun 30, 2020

exekias commented Jun 30, 2020

Labels

5 participants

	title: Kubernetes apiserver metrics
	title: Kubernetes API Server metrics

		title: Kubernetes apiserver metrics
		description: Collect Kubernetes apiserver metrics


		This integration is used to collect metrics from
		[Kubernetes clusters](https://kubernetes.io/).

		@@ -0,0 +1,44 @@
		title: Kubernetes Node metrics
		release: experimental

Uh oh!

Conversation

ChrsMark commented Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Migration Steps:

Screenshots

Manual Testing

elasticmachine commented Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

ChrsMark commented Jun 24, 2020

mtojek commented Jun 24, 2020

exekias left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrsMark commented Jun 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek commented Jun 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ChrsMark commented Jun 26, 2020

mtojek left a comment

Choose a reason for hiding this comment

ChrsMark commented Jun 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mtojek commented Jun 30, 2020

exekias commented Jun 30, 2020

Labels

5 participants

ChrsMark commented Jun 11, 2020 •

edited

Loading

elasticmachine commented Jun 11, 2020 •

edited

Loading

mtojek commented Jun 26, 2020 •

edited

Loading

ChrsMark commented Jun 26, 2020 •

edited

Loading