Skip to content

Enhancement: Add visualizations for data tier capacity - #10937

Merged
cp-elastic merged 14 commits into
mainfrom
elasticsearch/add-capacity-to-cluster-overview
Oct 21, 2024
Merged

Enhancement: Add visualizations for data tier capacity#10937
cp-elastic merged 14 commits into
mainfrom
elasticsearch/add-capacity-to-cluster-overview

Conversation

@cp-elastic

@cp-elastic cp-elastic commented Aug 29, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

Add visualizations for data tier capacity

Why

Based on feedback from my customers, these visualizations have been incredibly helpful for capacity planning and adjusting ILM policies.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Screenshots

capacity overview
@cp-elastic cp-elastic changed the title Elasticsearch/add capacity to cluster overview Aug 29, 2024
@andrewkroh andrewkroh added Integration:elasticsearch Elasticsearch enhancement New feature or request labels Aug 29, 2024
@elasticmachine

elasticmachine commented Aug 29, 2024 •
edited
Loading

Copy link
Copy Markdown

🚀 Benchmarks report

Package elasticsearch 👍(0) 💚(3) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
deprecation 6802.72 4878.05 -1924.67 (-28.29%) 💔
slowlog 5649.72 3164.56 -2485.16 (-43.99%) 💔

To see the full report comment with /test benchmark fullreport
@cp-elastic
cp-elastic marked this pull request as ready for review August 29, 2024 18:38
@cp-elastic
cp-elastic requested a review from a team as a code owner August 29, 2024 18:38
@andrewkroh andrewkroh added the Team:Stack Monitoring Stack Monitoring team [elastic/stack-monitoring] label Aug 29, 2024
@consulthys

consulthys commented Aug 30, 2024

Copy link
Copy Markdown
Contributor

@cp-elastic thank you for this contribution!
Can you please enumerate the required steps to test this PR?
@andrewkroh andrewkroh added the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Aug 30, 2024
@cp-elastic

cp-elastic commented Sep 3, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@consulthys
I'm not sure what the process is to build this integration locally for development. I exported the dashboard as .ndjson for testing on a ESS deployment.
ETA: Testing the dashboard iteself, gather Stack Monitoring Metrics via Beats or Agent, or on ESS > Ship to Deployment, and these graphs will track used vs free space per tier with an 80% Low Watermark line.
@consulthys

consulthys commented Sep 3, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Thank you @cp-elastic
I tried to import the dashboard into Kibana but it didn't work out, probably because the format is not the expected one (i.e. the file in this PR is a json one, not ndjson).
@cp-elastic

cp-elastic commented Sep 3, 2024

Copy link
Copy Markdown
Contributor Author

Updated the dashboards by exporting them properly. Tested against a fresh deployment, and is working as expected out of the box.
@cp-elastic

cp-elastic commented Sep 3, 2024

Copy link
Copy Markdown
Contributor Author

Ok, changed TSVB to Lens to fall in line with best practices guidance.
Updated the screenshot to reflect the new Lens visualizations.
@elastic-sonarqube

elastic-sonarqube Bot commented Sep 3, 2024

Copy link
Copy Markdown

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube
@elasticmachine

elasticmachine commented Sep 3, 2024

Copy link
Copy Markdown

💚 Build Succeeded

History
@botelastic

botelastic Bot commented Oct 3, 2024

Copy link
Copy Markdown

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!
@botelastic botelastic Bot added the Stalled label Oct 3, 2024
consulthys
consulthys approved these changes Oct 21, 2024
View reviewed changes

@consulthys consulthys left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGT Stack Monitoring
PS: sorry for the delay
@botelastic botelastic Bot removed the Stalled label Oct 21, 2024
@cp-elastic
cp-elastic merged commit da39aaf into main Oct 21, 2024
@cp-elastic
cp-elastic deleted the elasticsearch/add-capacity-to-cluster-overview branch October 21, 2024 11:57
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Oct 21, 2024

Copy link
Copy Markdown
Contributor

Package elasticsearch - 1.15.1 containing this change is available at https://epr.elastic.co/search?package=elasticsearch
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@cp-elastic
Enhancement: Add visualizations for data tier capacity (elastic#10937)
fca0e74 
* Rename Desconnections to Disconnections

* Adds 4 visualtions to show capacity per data tier

* Undo change from other branch

* Really undo changes from other branch

* Update changelog, add screenshot of capacity overview

* Increment version in manifest.yml

* Update capcity overview screenshot

* Update manifest.yml with screenshot

* Fix dashboard export

* Revert "typeMigrationVersion", remove "updated_by"

* Change TSVB to Lens

* remove showApplySelections

* Update screenshot showing Lens and manifest.yml

* Fix Data Warm filter from Hot
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@cp-elastic
Enhancement: Add visualizations for data tier capacity (elastic#10937)
ede1427 
* Rename Desconnections to Disconnections

* Adds 4 visualtions to show capacity per data tier

* Undo change from other branch

* Really undo changes from other branch

* Update changelog, add screenshot of capacity overview

* Increment version in manifest.yml

* Update capcity overview screenshot

* Update manifest.yml with screenshot

* Fix dashboard export

* Revert "typeMigrationVersion", remove "updated_by"

* Change TSVB to Lens

* remove showApplySelections

* Update screenshot showing Lens and manifest.yml

* Fix Data Warm filter from Hot
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dashboard Relates to a Kibana dashboard bug, enhancement, or modification. enhancement New feature or request Integration:elasticsearch Elasticsearch Team:Stack Monitoring Stack Monitoring team [elastic/stack-monitoring]

4 participants

@cp-elastic @elasticmachine @consulthys @andrewkroh