Add documentation for k8s alerts installation by ChrsMark · Pull Request #5364 · elastic/integrations

ChrsMark · 2023-02-23T03:52:15Z

This PR adds basic alert samples for Kubernetes clusters. The alerts for now are suggested inside the documentation pages but the long term solution would be to add them as part of the package's assets so that they can be installed through Kibana/Fleet.

More details and the related discussions can be found at #4997.

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

elasticmachine · 2023-02-23T04:03:35Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-03-01T08:41:00.853+0000
Duration: 29 min 26 sec

Test stats 🧪

Test	Results
Failed	0
Passed	92
Skipped	0
Total	92

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

elasticmachine · 2023-02-28T04:56:48Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`0/0`)	💚
Files	100.0% (`0/0`)	💚
Classes	100.0% (`0/0`)	💚
Methods	96.154% (`75/78`)	👎 -3.846
Lines	100.0% (`0/0`)	💚 5.682
Conditionals	100.0% (`0/0`)	💚

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

MichaelKatsoulis · 2023-02-28T08:43:14Z

+                    "must": [
+                      {
+                        "query_string": {
+                          "query": "event.module:kubernetes AND metricset.name: apiserver AND NOT (kubernetes.apiserver.request.verb: WATCH or kubernetes.apiserver.request.verb: CONNECT)",


Instead of event.module:kubernetes AND metricset.name: apiserver we can use data_stream.dataset: kubernetes.apiserver the same way we do in the dashboards.

This comment applies for all.

MichaelKatsoulis · 2023-02-28T08:53:17Z

How can a user access these alerts suggestions? Should we add a link in the kubernetes integration README ?

ChrsMark · 2023-02-28T09:33:16Z

How can a user access these alerts suggestions? Should we add a link in the kubernetes integration README ?

Those would be listed at https://docs.elastic.co/integrations/kubernetes like the docs' pages for the data_streams families. I can check if I can link them somehow but maybe the README is not a good place? Maybe https://www.elastic.co/guide/en/fleet/master/running-on-kubernetes-managed-by-fleet.html or a similar page is a better place to put them?
@gizas any thoughts here? Personally I would be ok with not advertising those much since the long term plan is to make them part of the package as assets. The current approach is more of a validation/PoC.

gizas · 2023-02-28T12:52:26Z

Wondering if writing a blog can be an option to advertise the ongoing work
We also have the work of automation through CI where same comments were received. That we have something that users dont know.

If we go for a blog a new story should be created.
Also opening a ticket to docs team and include such details for above stories can also be an option

ChrsMark · 2023-03-01T03:05:54Z

Wondering if writing a blog can be an option to advertise the ongoing work We also have the work of automation through CI where same comments were received. That we have something that users dont know.

If we go for a blog a new story should be created. Also opening a ticket to docs team and include such details for above stories can also be an option

Yes a blogpost is planned for this as listed in #4997.
In this regard, I would suggest to re-evaluate the docs coverage once we have completed #4997 so as to have the big picture of what is missing and what is the best way to cover it.
@MichaelKatsoulis @gizas let me know what do you think folks.

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

gizas · 2023-03-01T08:00:16Z

+                  "aggs": {
+                    "my_rate": {
+                      "rate": {
+                        "field": "kubernetes.pod.network.rx.errors",


Suggested change

"field": "kubernetes.pod.network.rx.errors",

"field": "kubernetes.pod.network.tx.errors",

gizas · 2023-03-01T08:00:31Z

+  },
+  "condition": {
+    "array_compare": {
+      "ctx.payload.aggregations.rx_error_rates.buckets": {


Suggested change

"ctx.payload.aggregations.rx_error_rates.buckets": {

"ctx.payload.aggregations.tx_error_rates.buckets": {

gizas · 2023-03-01T08:00:54Z

+  },
+  "actions": {
+    "log_hits": {
+      "foreach": "ctx.payload.aggregations.rx_error_rates.buckets",


Suggested change

"foreach": "ctx.payload.aggregations.rx_error_rates.buckets",

"foreach": "ctx.payload.aggregations.tx_error_rates.buckets",

gizas · 2023-03-01T08:01:13Z

+      "foreach": "ctx.payload.aggregations.rx_error_rates.buckets",
+      "max_iterations": 500,
+      "logging": {
+        "text": "Kubernetes Pod found with high rx error rate: {{ctx.payload.key}} -> {{ctx.payload.avg_minute_rate.value}}"


Suggested change

"text": "Kubernetes Pod found with high rx error rate: {{ctx.payload.key}} -> {{ctx.payload.avg_minute_rate.value}}"

"text": "Kubernetes Pod found with high tx error rate: {{ctx.payload.key}} -> {{ctx.payload.avg_minute_rate.value}}"

gizas · 2023-03-01T08:01:24Z

+    "xpack": {
+      "type": "json"
+    },
+    "name": "Pod RX Error Rate"


Suggested change

"name": "Pod RX Error Rate"

"name": "Pod TX Error Rate"

gizas · 2023-03-01T08:02:08Z

@@ -0,0 +1,102 @@
+## PID Pressure Kubernetes Node alert
+
+In order to install the PID Pressure Node alert run the following:


Can you provide a link for PID Pressure explanation?

gizas · 2023-03-01T08:02:54Z

@@ -0,0 +1,79 @@
+## Kubernetes Kubernetes Shipped Docs alert
+
+In order to install the Kubernetes Shipped Docs alert run the following:


Can you provide one sentence why we need this alert?

gizas · 2023-03-01T08:03:25Z

@@ -0,0 +1,102 @@
+## Unschedulable Kubernetes Node alert
+
+In order to install the Unschedulable Node alert run the following:


Also a link for what unsceduble nodes are can be handy

MichaelKatsoulis · 2023-03-01T08:10:06Z

In this regard, I would suggest to re-evaluate the docs coverage once we have completed #4997 so as to have the big picture of what is missing and what is the best way to cover it.

@ChrsMark I agree about the blogspot. Regarding opening a PR to the docs team to include those alerts in the official documentation I would expect this to be also listed in the #4997 so that it can be tracked.

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark · 2023-03-01T08:31:19Z

@gizas @MichaelKatsoulis thanks for the feedback folks! I think your comments are now covered. Feel free to have another look when you have the time.

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

elasticmachine · 2023-03-01T10:59:29Z

Package kubernetes - 1.32.0 containing this change is available at https://epr.elastic.co/search?package=kubernetes

Add 1st alert

abd936c

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark added 2 commits February 24, 2023 12:03

Move alert into docs

234f5f5

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Add all alerts under docs

9826c50

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark changed the title ~~Add 1st alert~~ Feb 28, 2023

ChrsMark requested a review from gizas February 28, 2023 04:15

ChrsMark self-assigned this Feb 28, 2023

ChrsMark marked this pull request as ready for review February 28, 2023 04:15

ChrsMark requested a review from a team as a code owner February 28, 2023 04:15

ChrsMark requested a review from constanca-m February 28, 2023 04:15

ChrsMark added 2 commits February 28, 2023 12:16

Merge remote-tracking branch 'upstream/main' into add_alerts

b1237d6

Add changelog and bump up version

f6ed32b

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

This was referenced Feb 28, 2023

Starting Kubernetes Observability through basic SLIs/SLOs #4997

Closed

Add alerts/watchers as assets in packages elastic/package-spec#484

Closed

gizas approved these changes Feb 28, 2023

View reviewed changes

Add name for API Server alert

fd358bb

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

MichaelKatsoulis reviewed Feb 28, 2023

View reviewed changes

Fix queries filters

bd7b857

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark requested review from MichaelKatsoulis and gizas March 1, 2023 03:10

gizas reviewed Mar 1, 2023

View reviewed changes

Comment thread packages/kubernetes/docs/pod-terminated-oomkilled-alert.md Outdated

gizas reviewed Mar 1, 2023

View reviewed changes

ChrsMark added 2 commits March 1, 2023 16:19

Review fixes

ae102cb

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Explain shipped docs alert

0115dbd

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark requested a review from gizas March 1, 2023 08:31

Fix filters to use data_stream field

e2d0d7f

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

MichaelKatsoulis approved these changes Mar 1, 2023

View reviewed changes

gizas approved these changes Mar 1, 2023

View reviewed changes

ChrsMark merged commit afea63c into elastic:main Mar 1, 2023

agithomas pushed a commit to agithomas/integrations that referenced this pull request Mar 20, 2023

Add documentation for k8s alerts installation (elastic#5364)

30b5af9

agithomas pushed a commit to agithomas/integrations that referenced this pull request Mar 21, 2023

Add documentation for k8s alerts installation (elastic#5364)

a5376f0

orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026

Add documentation for k8s alerts installation (elastic#5364)

27a9bdd

Navigation Menu

Uh oh!

Add documentation for k8s alerts installation - #5364

Add documentation for k8s alerts installation#5364
ChrsMark merged 10 commits into
elastic:mainfrom
ChrsMark:add_alerts

ChrsMark commented Feb 23, 2023 •

edited

Loading

elasticmachine commented Feb 23, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Feb 28, 2023 •

edited

Loading

MichaelKatsoulis Feb 28, 2023

MichaelKatsoulis Feb 28, 2023

MichaelKatsoulis commented Feb 28, 2023

ChrsMark commented Feb 28, 2023

gizas commented Feb 28, 2023

ChrsMark commented Mar 1, 2023 •

edited

Loading

Uh oh!

gizas Mar 1, 2023

gizas Mar 1, 2023

gizas Mar 1, 2023

gizas Mar 1, 2023

gizas Mar 1, 2023

gizas Mar 1, 2023

gizas Mar 1, 2023

gizas Mar 1, 2023

MichaelKatsoulis commented Mar 1, 2023

ChrsMark commented Mar 1, 2023

elasticmachine commented Mar 1, 2023

Labels

4 participants

	"field": "kubernetes.pod.network.rx.errors",
	"field": "kubernetes.pod.network.tx.errors",

	"ctx.payload.aggregations.rx_error_rates.buckets": {
	"ctx.payload.aggregations.tx_error_rates.buckets": {

	"foreach": "ctx.payload.aggregations.rx_error_rates.buckets",
	"foreach": "ctx.payload.aggregations.tx_error_rates.buckets",

	"text": "Kubernetes Pod found with high rx error rate: {{ctx.payload.key}} -> {{ctx.payload.avg_minute_rate.value}}"
	"text": "Kubernetes Pod found with high tx error rate: {{ctx.payload.key}} -> {{ctx.payload.avg_minute_rate.value}}"

		@@ -0,0 +1,102 @@
		## PID Pressure Kubernetes Node alert

		In order to install the PID Pressure Node alert run the following:

		@@ -0,0 +1,79 @@
		## Kubernetes Kubernetes Shipped Docs alert

		In order to install the Kubernetes Shipped Docs alert run the following:

		@@ -0,0 +1,102 @@
		## Unschedulable Kubernetes Node alert

		In order to install the Unschedulable Node alert run the following:

Uh oh!

Conversation

ChrsMark commented Feb 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

elasticmachine commented Feb 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Feb 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelKatsoulis commented Feb 28, 2023

ChrsMark commented Feb 28, 2023

gizas commented Feb 28, 2023

ChrsMark commented Mar 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelKatsoulis commented Mar 1, 2023

ChrsMark commented Mar 1, 2023

elasticmachine commented Mar 1, 2023

Labels

4 participants

ChrsMark commented Feb 23, 2023 •

edited

Loading

elasticmachine commented Feb 23, 2023 •

edited

Loading

elasticmachine commented Feb 28, 2023 •

edited

Loading

ChrsMark commented Mar 1, 2023 •

edited

Loading