Force rollover on write to true when data stream indices list is empty#133347
Merged
jbaiera merged 11 commits intoelastic:mainfrom Aug 22, 2025
Merged
Force rollover on write to true when data stream indices list is empty#133347jbaiera merged 11 commits intoelastic:mainfrom
jbaiera merged 11 commits intoelastic:mainfrom
Conversation
jbaiera
added
>bug
:StorageEngine/Data streams
elasticsearchmachine
added
the
Team:Data Management (obsolete)
Collaborator
|
Pinging @elastic/es-data-management (Team:Data Management) |
Collaborator
|
Hi @jbaiera, I've created a changelog YAML for you. |
samxbr
reviewed
Aug 21, 2025
qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/FailureStoreUpgradeIT.java
Outdated
Show resolved
Hide resolved
qa/rolling-upgrade/src/javaRestTest/java/org/elasticsearch/upgrades/FailureStoreUpgradeIT.java
Show resolved
Hide resolved
Member
Author
|
CI is failing because rollover on write is forced true on any data stream index set which is empty, but this is not allowed because data streams created for CCR may not have rollover on write enabled. As such, I have a fix coming in which should simply update the default places in deserialization that we set the flag and make them default to true on an empty index list. |
samxbr
approved these changes
Aug 21, 2025
| dataStreamOptions, | ||
| new DataStreamIndices(BACKING_INDEX_PREFIX, List.copyOf(indices), rolloverOnWrite, autoShardingEvent), | ||
| new DataStreamIndices(FAILURE_STORE_PREFIX, List.copyOf(failureIndices), false, null) | ||
| new DataStreamIndices(FAILURE_STORE_PREFIX, List.copyOf(failureIndices), failureIndices.isEmpty(), null) |
Contributor
There was a problem hiding this comment.
Is the reason we force rolloverOnWrite as a param to the next constructor here instead other constructor to bypass the CCR error you mentioned? If so I find it a little subtle to read, but I am not sure how to do it more explicitly.
Disregard this, I think I mis-read the constructor arguments, there are too many of them
jbaiera
force-pushed
the
fix-failure-store-upgrade
branch
from
390818d to
d7fbb12
Compare
Member
Author
💚 All backports created successfully
Questions ?Please refer to the Backport tool documentation |
jbaiera
added a commit
to jbaiera/elasticsearch
that referenced
this pull request
Aug 22, 2025
elastic#133347) Updates the serialization and construction logic for failure store components to bias rollover on write to true when no indices are present in the failure store index set. --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Keith Massey <keith.massey@elastic.co> (cherry picked from commit 5ae52bf)
pabloem
pushed a commit
to pabloem/elasticsearch
that referenced
this pull request
Aug 22, 2025
elastic#133347) Updates the serialization and construction logic for failure store components to bias rollover on write to true when no indices are present in the failure store index set. --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Keith Massey <keith.massey@elastic.co>
elasticsearchmachine
pushed a commit
that referenced
this pull request
Aug 22, 2025
#133347) (#133406) Updates the serialization and construction logic for failure store components to bias rollover on write to true when no indices are present in the failure store index set. --------- (cherry picked from commit 5ae52bf) Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Keith Massey <keith.massey@elastic.co>
phananh1010
added a commit
to phananh1010/elasticsearch
that referenced
this pull request
Aug 28, 2025
phananh1010
added a commit
to phananh1010/elasticsearch
that referenced
this pull request
Aug 28, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If a data stream indices list is ever empty, then it cannot accept any write traffic. This is a state that is not allowed in the normal backing indices, but as we add more index sets to a data stream that are lazily created like failure store, this property should not only reflect any state that was marked via a lazy rollover API call, but also be forced on when no indices exist so that they may be initialized on first write.
An upgrade test is present to ensure that failure store indices created on versions from before their support are able to be enabled and used.
Fixes #133176