Skip to content

[Kubernetes] Add state namespace datastream - #8329

Merged
constanca-m merged 10 commits into
elastic:mainfrom
constanca-m:add-state-namespace
Oct 31, 2023
Merged

[Kubernetes] Add state namespace datastream#8329
constanca-m merged 10 commits into
elastic:mainfrom
constanca-m:add-state-namespace

Conversation

@constanca-m

@constanca-m constanca-m commented Oct 30, 2023

Copy link
Copy Markdown
Contributor

What

Add state namespace data stream.

The metricset was added in version 8.11.0.

Dimensions for TSDB:

  • kubernetes.namespace is unique, so it is a dimension.
  • agent.id is a common dimension among data streams.
  • orchestrator.cluster.url is a dimension for all kubernetes data streams.
Results from running TSDB testing kit:
The time series fields for the TSDB index are: 
	- dimension (3 fields):
		- agent.id
		- kubernetes.namespace
		- orchestrator.cluster.url
	- gauge (1 fields):
		- kubernetes.state_namespace.created.sec
	- routing_path (3 fields):
		- agent.id
		- kubernetes.namespace
		- orchestrator.cluster.url

Index tsdb-index-enabled successfully created.

Copying documents from .ds-metrics-kubernetes.state_namespace-default-2023.10.30-000001 to tsdb-index-enabled...
All 210 documents taken from index .ds-metrics-kubernetes.state_namespace-default-2023.10.30-000001 were successfully placed to index tsdb-index-enabled.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  1. Clone this branch.
  2. Build the package.
  3. Update the service registry.
  4. Add Kubernetes to a policy and check the new data stream.

Related issues

Relates to elastic/elastic-agent#3100.

Screenshots

In Discover:
image

In the integration itself:
image

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
@constanca-m constanca-m added Integration:kubernetes Kubernetes Team:Cloudnative-Monitoring Cloud Native Monitoring team [elastic/obs-cloudnative-monitoring] labels Oct 30, 2023
@constanca-m constanca-m self-assigned this Oct 30, 2023
@constanca-m
constanca-m requested a review from a team as a code owner October 30, 2023 10:57
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
@elasticmachine

elasticmachine commented Oct 30, 2023

Copy link
Copy Markdown

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-10-31T16:41:46.770+0000

  • Duration: 77 min 5 sec

Test stats 🧪

Test Results
Failed 0
Passed 97
Skipped 0
Total 97

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine

elasticmachine commented Oct 30, 2023

Copy link
Copy Markdown

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (1/1) 💚
Files 100.0% (1/1) 💚
Classes 100.0% (1/1) 💚
Methods 96.386% (80/83)
Lines 100.0% (22/22) 💚
Conditionals 100.0% (0/0) 💚
Comment thread packages/kubernetes/data_stream/state_namespace/fields/fields.yml Outdated
constanca-m and others added 2 commits October 31, 2023 09:19
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
@gizas

gizas commented Oct 31, 2023

Copy link
Copy Markdown
Contributor

Another idea is to consider if we need to add this info of namespace in our dashboards. To add it in Overview or in Pods?

@MichaelKatsoulis , @tetianakravchenko , @ChrsMark suggestions?

Comment thread packages/kubernetes/data_stream/state_namespace/fields/agent.yml
Comment thread packages/kubernetes/data_stream/state_namespace/fields/agent.yml
Comment thread packages/kubernetes/data_stream/state_namespace/fields/base-fields.yml Outdated
Comment thread packages/kubernetes/data_stream/state_namespace/fields/base-fields.yml Outdated
Kubernetes namespace metrics.

fields:
- name: created.sec

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should here be defined metric_type ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since TSDB won't be enabled, then the metric type does not need to be defined.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would not be enabled?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The advantages of TSDB are related to the metric types. This data stream only has one metric type, so there wouldn't be any. We followed a similar approach for other data streams that only had one metric, such as AWS Billing or GCP Billing @gizas

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After TSDB meeting discussion, we are enabling TSDB in all metric data streams after all. I updated the commits and will update the description now.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Comment thread packages/kubernetes/manifest.yml
Enable TSDB.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
@constanca-m
constanca-m merged commit ca6dc49 into elastic:main Oct 31, 2023
@constanca-m
constanca-m deleted the add-state-namespace branch October 31, 2023 19:00
@elasticmachine

Copy link
Copy Markdown

Package kubernetes - 1.52.0 containing this change is available at https://epr.elastic.co/search?package=kubernetes

@elasticmachine

Copy link
Copy Markdown

Package kubernetes - 1.55.0 containing this change is available at https://epr.elastic.co/search?package=kubernetes

orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
* Add state namespace

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Update changelog.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Update description.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Delete dimensions.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Delete extra base fields.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Add dimensions and metric type.

Enable TSDB.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Update KSM read me file.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Update KSM read me file.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

---------

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Integration:kubernetes Kubernetes Team:Cloudnative-Monitoring Cloud Native Monitoring team [elastic/obs-cloudnative-monitoring]

4 participants