Skip to content

[RedisEnterprise] Enable use_type flag for node and proxy datastreams #8213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ritalwar merged 13 commits into from
Nov 15, 2023
Merged

[RedisEnterprise] Enable use_type flag for node and proxy datastreams #8213

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
2be582a
[RedisEnterprise] Enable use_type flag for node and proxy datastreams.
ritalwar Oct 16, 2023
3d188f3
Update changelog.yml
ritalwar Oct 16, 2023
a61b7c1
Update Dashboards
ritalwar Oct 18, 2023
c9bb735
Update Dashboard title
ritalwar Oct 18, 2023
cbc22d1
Update README.md
ritalwar Oct 26, 2023
e0f92de
Merge branch 'main' into redisenterprise_usetype_8149
ritalwar Oct 26, 2023
85cd600
Add object_type
ritalwar Oct 26, 2023
df0b19f
Update sample_event.json
ritalwar Oct 26, 2023
d559c67
Update fields mappings for node and proxy datastreams
ritalwar Oct 27, 2023
e1d8909
Update changelog.yml
ritalwar Nov 2, 2023
2553c00
Address review comments
ritalwar Nov 9, 2023
0d60e7c
Address comments
ritalwar Nov 10, 2023
1d5961d
Update fields.yml
ritalwar Nov 15, 2023
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion packages/redisenterprise/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

Redis Enterprise integration provides monitoring of [redis](https://redis.com/) cluster. Monitoring is done via prometheus exported port of redis enterprise cluster. Once a redis enterprise [cluster](https://redis.com/redis-enterprise/technology/redis-enterprise-cluster-architecture/) is installed, prometheus port is available for monitoring. The url of the host:port(8070) needs to be passed to the hosts in the settings.

Redis Enterpise integration is tested with redislabs/redis:5.2.2-24 version.
## Compatibility
Tested with Redis Enterprise v7.2.4.

# Metrics

Expand Down
5 changes: 5 additions & 0 deletions packages/redisenterprise/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "0.7.0"
changes:
- description: Enable use_types flag for node and proxy datastreams.
type: enhancement
link: https://github.com/elastic/integrations/pull/8213
- version: 0.6.0
changes:
- description: Update the package format_version to 3.0.0.
Expand Down
6 changes: 1 addition & 5 deletions packages/redisenterprise/data_stream/node/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
metricsets: ["collector"]
period: {{period}}

# Disable
use_types: false
rate_counters: false
Comment thread
ritalwar marked this conversation as resolved.

hosts:
{{#each hosts}}
- {{this}}
Expand All @@ -18,4 +14,4 @@ metrics_filters.exclude: ["^up$"]
metrics_path:

ssl.verification_mode: "none"

use_types: true
2 changes: 1 addition & 1 deletion packages/redisenterprise/data_stream/node/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ processors:
ignore_missing: true
ignore_failure: true
- foreach:
field: redisenterprise.node.metrics
field: redisenterprise.node
processor:
gsub:
field: "_ingest._key"
Expand Down
21 changes: 10 additions & 11 deletions packages/redisenterprise/data_stream/node/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
- name: redisenterprise.node
type: group
fields:
- name: metrics.*
# use pattern for matching to ensure future extensibillity.
type: float
description: Node prometheus metrics
- name: labels.*
type: object
object_type: keyword
description: Label fields
- name: redisenterprise.node.*.value

@lalit-satapathy lalit-satapathy Nov 2, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming the metric_type mapping will be taken in separate PR.

@ritalwar ritalwar Nov 2, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, those will be raised separately as part of the TSDB migration.

@lalit-satapathy lalit-satapathy Nov 6, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming the metric type mappsing is: redisenterprise.node.*.value will directly map to gauge and redisenterprise.proxy.*.counter to counter.

@ritalwar ritalwar Nov 9, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we do need to explicitly specify the metric_type, and I will address this in the metric_type PR.
type: object
object_type: double
object_type_mapping_type: "*"
description: Node metrics.

- name: redisenterprise.node.labels.*
Comment thread
ritalwar marked this conversation as resolved.
type: object
object_type: keyword
description: Label fields
177 changes: 109 additions & 68 deletions packages/redisenterprise/data_stream/node/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,12 @@
{
"agent": {
"name": "docker-fleet-agent",
"id": "56370311-c973-4a5b-b93b-e42fd47da568",
"ephemeral_id": "5d1146d0-7ec9-44be-baa0-3aefc54e6ffd",
"id": "8412e9e5-6fd3-4587-bc60-1fba76200570",
"type": "metricbeat",
"version": "8.3.0"
"ephemeral_id": "cc4072b4-71a6-40be-ad91-7245b283f11d",
"version": "8.10.4"
},
"elastic_agent": {
"id": "56370311-c973-4a5b-b93b-e42fd47da568",
"version": "8.3.0",
"snapshot": true
},
"@timestamp": "2022-07-12T07:33:25.716Z",
"@timestamp": "2023-10-27T08:01:51.865Z",
"ecs": {
"version": "8.5.1"
},
Expand All @@ -30,78 +25,124 @@
"kernel": "5.10.104-linuxkit",
"codename": "focal",
"name": "Ubuntu",
"family": "debian",
"type": "linux",
"version": "20.04.4 LTS (Focal Fossa)",
"family": "debian",
"version": "20.04.6 LTS (Focal Fossa)",
"platform": "ubuntu"
},
"containerized": false,
"ip": [
"172.21.0.7"
"172.19.0.7"
],
"containerized": false,
"name": "docker-fleet-agent",
"id": "5bf910baf91142d6b435357818c88ef5",
"mac": [
"02:42:ac:15:00:07"
"02-42-AC-13-00-07"
],
"architecture": "x86_64"
"architecture": "aarch64"
},
"elastic_agent": {
"id": "8412e9e5-6fd3-4587-bc60-1fba76200570",
"version": "8.10.4",
"snapshot": false
},
"redisenterprise": {
"node": {
"metrics": {
"provisional_memory": 7075143156.111,
"cpu_nice_median": 0,
"cpu_system_min": 0.081,
"cpu_irqs_max": 0.003,
"persistent_storage_free": 224708261372814.22,
"cpu_user_min": 0.528,
"egress_bytes_median": 9161.667,
"cpu_user_median": 0.533,
"cpu_iowait_median": 0.001,
"ephemeral_storage_free": 195630087281.778,
"egress_bytes_min": 620.556,
"cpu_steal_median": 0,
"cpu_idle_min": 0.101,
"cpu_irqs_median": 0.002,
"ingress_bytes_min": 622.667,
"provisional_memory_no_overbooking": 7073454693.889,
"conns": 0,
"cpu_iowait_max": 0.001,
"cpu_steal_min": 0,
"cpu_steal": 0,
"cpu_iowait": 0.0006666666666666666,
"cpu_irqs": 0.0023333333333333335,
"egress_bytes_max": 18004,
"cpu_user": 0.6206666666666667,
"free_memory": 10688636245.333,
"ingress_bytes_max": 1340,
"available_memory_no_overbooking": 10632216778.889,
"ingress_bytes_median": 996.444,
"cpu_idle": 0.2823333333333333,
"cpu_nice_max": 0,
"cpu_system": 0.08666666666666667,
"cpu_idle_median": 0.373,
"cpu_iowait_min": 0,
"cpu_nice_min": 0,
"cur_aof_rewrites": 0,
"cpu_system_max": 0.092,
"persistent_storage_avail": 224708261372814.22,
"cpu_nice": 0,
"cpu_steal_max": 0,
"cpu_irqs_min": 0.002,
"cpu_idle_max": 0.373,
"available_memory": 10633905241.111,
"ingress_bytes": 986.3703333333333,
"ephemeral_storage_avail": 184875945073.778,
"egress_bytes": 9262.074333333332,
"cpu_user_max": 0.801,
"cpu_system_median": 0.087,
"total_req": 0
},
"labels": {
"cluster": "run1.local",
"cluster": "cluster.local",
"node": "1",
"instance": "host.docker.internal:8070",
"job": "prometheus"
},

@lalit-satapathy lalit-satapathy Nov 2, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a coincidence that all node* metrics are gauge/value?

@ritalwar ritalwar Nov 2, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've already done the metric_type PR for this datastream and verified then there that all the node datastream metrics are gauges. Also please refer.
"cpu_iowait_min": {
"value": 0.001
},
"cpu_nice_min": {
"value": 0
},
"cur_aof_rewrites": {
"value": 0
},
"cpu_system_max": {
"value": 0.037
},
"persistent_storage_avail": {
"value": 50210169287.111
},
"cpu_irqs_min": {
"value": 0.003
},
"cpu_idle_max": {
"value": 0.751
},
"ephemeral_storage_avail": {
"value": 50210169287.111
},
"egress_bytes": {
"value": 1342.5556666666669
},
"cpu_system_median": {
"value": 0.036
},
"total_req": {
"value": 0
},
"cpu_system_min": {
"value": 0.034
},
"persistent_storage_free": {
"value": 53426885063.111
},
"cpu_user_min": {
"value": 0.192
},
"egress_bytes_median": {
"value": 1569.778
},
"cpu_user_median": {
"value": 0.195
},
"cpu_steal_median": {
"value": 0
},
"egress_bytes_min": {
"value": 885.889
},
"ingress_bytes_min": {
"value": 242
},
"conns": {
"value": 0
},
"cpu_steal_min": {
"value": 0
},
"cpu_user": {
"value": 0.19533333333333336
},
"free_memory": {
"value": 257662065.778
},
"cpu_idle": {
"value": 0.7493333333333334
},
"cpu_system": {
"value": 0.03566666666666667
},
"cpu_steal_max": {
"value": 0
},
"cpu_nice": {
"value": 0
},
"available_memory": {
"value": 146729029.556
},
"ingress_bytes": {
"value": 258.815
},
"cpu_user_max": {
"value": 0.199
}
}
},
Expand All @@ -110,9 +151,9 @@
"name": "collector"
},
"event": {
"duration": 182194083,
"duration": 240631792,
"agent_id_status": "verified",
"ingested": "2022-07-12T07:33:26Z",
"ingested": "2023-10-27T08:01:52Z",
"module": "prometheus",
"dataset": "redisenterprise.node"
}
Expand Down
5 changes: 1 addition & 4 deletions packages/redisenterprise/data_stream/proxy/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
metricsets: ["collector"]
period: {{period}}

# disable
use_types: false
rate_counters: false

hosts:
{{#each hosts}}
- {{this}}
Expand All @@ -18,3 +14,4 @@ metrics_filters.exclude: ["^up$"]
metrics_path:

ssl.verification_mode: "none"
use_types: true
21 changes: 10 additions & 11 deletions packages/redisenterprise/data_stream/proxy/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
- name: redisenterprise.proxy
type: group
fields:
- name: metrics.listener_*
# use pattern for matching to ensure future extensibillity.
type: float
description: Proxy prometheus metrics
- name: labels.*
type: object
object_type: keyword
description: Label fields
- name: redisenterprise.proxy.*.counter
type: object
object_type: double
object_type_mapping_type: "*"
description: Proxy metrics.

- name: redisenterprise.proxy.labels.*
type: object
object_type: keyword
description: Label fields
Loading