Conversation
mattkime
commented
Nov 22, 2025
| nodeId: '9OFkjpAKS_aPzJAuEOSg7w', | ||
| nodeName: 'MacBook-Pro.local', | ||
| available: '25%', | ||
| lowDiskWatermarkSetting: '50%', |
Contributor
Author
There was a problem hiding this comment.
This value isn't displayed so I've removed it.
mattkime
added
Team:Kibana Management
Contributor
|
Pinging @elastic/kibana-management (Team:Kibana Management) |
mattkime
added
the
backport:all-open
sabarasaba
reviewed
Dec 8, 2025
Member
sabarasaba
left a comment
sabarasaba
left a comment
There was a problem hiding this comment.
Good stuff! happy we get to remove so much code from UA 🥳 Left a small comment let me know what you think
| }); | ||
|
|
||
| return response.ok({ body: nodesWithLowDiskSpace }); | ||
| if (!lowDiskWatermarkSetting) { |
Member
There was a problem hiding this comment.
Since we now rely on low_watermark_free_space_in_bytes from nodes.stats, we could still detect low disk conditions even if the watermark was set in elasticsearch.yml and not returned by cluster.get_settings. Should this guard be a bit more relaxed then or does it still make sense to keep? 🤔
sabarasaba
approved these changes
Dec 10, 2025
Contributor
💚 Build Succeeded
Metrics [docs]
History
cc @mattkime |
mattkime
added
v8.19.9
backport:version
mattkime
deleted the
upgrade_assistant_better_warning_on_low_watermark
branch
Contributor
|
Starting backport for target branches: 8.19 |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Dec 10, 2025
… watermark warning (elastic#243906) ## Summary Upgrade assistant shows a warning when a node exceeds its storage low watermark. However, the low watermark setting can be overridden by the max headroom setting which upgrade assistant fails to account for. Previously upgrade assistant was calculating the low watermark setting and failing to include the max headroom setting. Now the node stats api provides the calculated low watermark setting and much less calculation is performed. The amount of free space will now be shown in bytes (human readable format) instead of as a percentage. Modify cluster settings like so - ``` PUT _cluster/settings { "persistent": { "cluster.routing.allocation.disk.watermark.low": "85%", "cluster.routing.allocation.disk.watermark.flood_stage.max_headroom" : "998Gb", "cluster.routing.allocation.disk.watermark.high.max_headroom": "999Gb" } } ``` Follow directions to simulate a reindexing case - elastic#241222 Closes elastic#240002 ### Release Notes Fixes case where upgrade assistant would incorrectly warn about node breaching low watermark despite max headroom setting. (cherry picked from commit 39d9782)
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Dec 10, 2025
…ing low watermark warning (#243906) (#245876) # Backport This will backport the following commits from `main` to `8.19`: - [[upgrade assistant] Account for max headroom setting when showing low watermark warning (#243906)](#243906) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Matthew Kime","email":"matt@mattki.me"},"sourceCommit":{"committedDate":"2025-12-10T16:19:23Z","message":"[upgrade assistant] Account for max headroom setting when showing low watermark warning (#243906)\n\n## Summary\n\nUpgrade assistant shows a warning when a node exceeds its storage low\nwatermark. However, the low watermark setting can be overridden by the\nmax headroom setting which upgrade assistant fails to account for.\n\nPreviously upgrade assistant was calculating the low watermark setting\nand failing to include the max headroom setting. Now the node stats api\nprovides the calculated low watermark setting and much less calculation\nis performed.\n\nThe amount of free space will now be shown in bytes (human readable\nformat) instead of as a percentage.\n\nModify cluster settings like so - \n```\nPUT _cluster/settings\n{\n \"persistent\": {\n \"cluster.routing.allocation.disk.watermark.low\": \"85%\",\n \"cluster.routing.allocation.disk.watermark.flood_stage.max_headroom\" : \"998Gb\",\n \"cluster.routing.allocation.disk.watermark.high.max_headroom\": \"999Gb\"\n }\n}\n```\n\nFollow directions to simulate a reindexing case -\nhttps://github.com//pull/241222\n\n\nCloses https://github.com/elastic/kibana/issues/240002\n\n### Release Notes\n\nFixes case where upgrade assistant would incorrectly warn about node\nbreaching low watermark despite max headroom setting.","sha":"39d97825d665d7f10b262c5a4a1ee55043539a4e","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","Team:Kibana Management","Feature:Upgrade Assistant","backport:version","v9.3.0","v8.19.9"],"title":"[upgrade assistant] Account for max headroom setting when showing low watermark warning","number":243906,"url":"https://github.com/elastic/kibana/pull/243906","mergeCommit":{"message":"[upgrade assistant] Account for max headroom setting when showing low watermark warning (#243906)\n\n## Summary\n\nUpgrade assistant shows a warning when a node exceeds its storage low\nwatermark. However, the low watermark setting can be overridden by the\nmax headroom setting which upgrade assistant fails to account for.\n\nPreviously upgrade assistant was calculating the low watermark setting\nand failing to include the max headroom setting. Now the node stats api\nprovides the calculated low watermark setting and much less calculation\nis performed.\n\nThe amount of free space will now be shown in bytes (human readable\nformat) instead of as a percentage.\n\nModify cluster settings like so - \n```\nPUT _cluster/settings\n{\n \"persistent\": {\n \"cluster.routing.allocation.disk.watermark.low\": \"85%\",\n \"cluster.routing.allocation.disk.watermark.flood_stage.max_headroom\" : \"998Gb\",\n \"cluster.routing.allocation.disk.watermark.high.max_headroom\": \"999Gb\"\n }\n}\n```\n\nFollow directions to simulate a reindexing case -\nhttps://github.com//pull/241222\n\n\nCloses https://github.com/elastic/kibana/issues/240002\n\n### Release Notes\n\nFixes case where upgrade assistant would incorrectly warn about node\nbreaching low watermark despite max headroom setting.","sha":"39d97825d665d7f10b262c5a4a1ee55043539a4e"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/243906","number":243906,"mergeCommit":{"message":"[upgrade assistant] Account for max headroom setting when showing low watermark warning (#243906)\n\n## Summary\n\nUpgrade assistant shows a warning when a node exceeds its storage low\nwatermark. However, the low watermark setting can be overridden by the\nmax headroom setting which upgrade assistant fails to account for.\n\nPreviously upgrade assistant was calculating the low watermark setting\nand failing to include the max headroom setting. Now the node stats api\nprovides the calculated low watermark setting and much less calculation\nis performed.\n\nThe amount of free space will now be shown in bytes (human readable\nformat) instead of as a percentage.\n\nModify cluster settings like so - \n```\nPUT _cluster/settings\n{\n \"persistent\": {\n \"cluster.routing.allocation.disk.watermark.low\": \"85%\",\n \"cluster.routing.allocation.disk.watermark.flood_stage.max_headroom\" : \"998Gb\",\n \"cluster.routing.allocation.disk.watermark.high.max_headroom\": \"999Gb\"\n }\n}\n```\n\nFollow directions to simulate a reindexing case -\nhttps://github.com//pull/241222\n\n\nCloses https://github.com/elastic/kibana/issues/240002\n\n### Release Notes\n\nFixes case where upgrade assistant would incorrectly warn about node\nbreaching low watermark despite max headroom setting.","sha":"39d97825d665d7f10b262c5a4a1ee55043539a4e"}},{"branch":"8.19","label":"v8.19.9","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Matthew Kime <matt@mattki.me>
seanrathier
pushed a commit
to seanrathier/kibana
that referenced
this pull request
Dec 15, 2025
… watermark warning (elastic#243906) ## Summary Upgrade assistant shows a warning when a node exceeds its storage low watermark. However, the low watermark setting can be overridden by the max headroom setting which upgrade assistant fails to account for. Previously upgrade assistant was calculating the low watermark setting and failing to include the max headroom setting. Now the node stats api provides the calculated low watermark setting and much less calculation is performed. The amount of free space will now be shown in bytes (human readable format) instead of as a percentage. Modify cluster settings like so - ``` PUT _cluster/settings { "persistent": { "cluster.routing.allocation.disk.watermark.low": "85%", "cluster.routing.allocation.disk.watermark.flood_stage.max_headroom" : "998Gb", "cluster.routing.allocation.disk.watermark.high.max_headroom": "999Gb" } } ``` Follow directions to simulate a reindexing case - elastic#241222 Closes elastic#240002 ### Release Notes Fixes case where upgrade assistant would incorrectly warn about node breaching low watermark despite max headroom setting.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrade assistant shows a warning when a node exceeds its storage low watermark. However, the low watermark setting can be overridden by the max headroom setting which upgrade assistant fails to account for.
Previously upgrade assistant was calculating the low watermark setting and failing to include the max headroom setting. Now the node stats api provides the calculated low watermark setting and much less calculation is performed.
The amount of free space will now be shown in bytes (human readable format) instead of as a percentage.
Modify cluster settings like so -
Follow directions to simulate a reindexing case - #241222
Closes #240002
Release Notes
Fixes case where upgrade assistant would incorrectly warn about node breaching low watermark despite max headroom setting.