Retry ILM async action after reindexing data stream by parkertimmins · Pull Request #124149 · elastic/elasticsearch

parkertimmins · 2025-03-05T20:36:51Z

When reindexing a data stream, the ILM metadata is copied from the index metadata of the source index to the destination index. But the ILM state of the new index can be stuck if the source index was in an AsyncAction at the time of reindexing. To un-stick the new index, we call TransportRetryAction to retry the AsyncAction. In the past this action would only run if the index were in the error phase. This change includes an update to TransportRetryAction, which allows it to be run when the index is not in an error phase, if the parameter requireError is set to false.

elasticsearchmachine · 2025-03-05T20:44:38Z

Pinging @elastic/es-data-management (Team:Data Management)

masseyke · 2025-03-05T21:22:19Z

.../main/java/org/elasticsearch/xpack/migrate/task/ReindexDataStreamPersistentTaskExecutor.java

        }));
    }

+    private void maybeRunILMAsyncAction(String newIndex, ActionListener<AcknowledgedResponse> listener, TaskId parentTaskId) {


It would be good to have a comment explaining why we have to do this.

masseyke

LGTM

elasticsearchmachine · 2025-03-06T17:21:21Z

Hi @parkertimmins, I've created a changelog YAML for you.

elasticsearchmachine · 2025-03-06T18:41:09Z

💔 Backport failed

Status	Branch	Result
❌	9.0	Commit could not be cherrypicked due to conflicts
❌	8.18	Commit could not be cherrypicked due to conflicts
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 124149

When reindexing a data stream, the ILM metadata is copied from the index metadata of the source index to the destination index. But the ILM state of the new index can be stuck if the source index was in an AsyncAction at the time of reindexing. To un-stick the new index, we call TransportRetryAction to retry the AsyncAction. In the past this action would only run if the index were in the error phase. This change includes an update to TransportRetryAction, which allows it to be run when the index is not in an error phase, if the parameter requireError is set to false. (cherry picked from commit 10a8dcf) # Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java # x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/user/InternalUsers.java # x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/action/TransportRetryAction.java # x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/DataStreamsUpgradeIT.java

parkertimmins · 2025-03-06T18:57:53Z

💔 Some backports could not be created

Status	Branch	Result
✅	8.x
❌	9.0	Conflict resolution was aborted by the user
❌	8.18	Conflict resolution was aborted by the user

Manual backport

To create the backport manually run:

backport --pr 124149

Questions ?

Please refer to the Backport tool documentation

parkertimmins · 2025-03-06T19:46:44Z

💔 Some backports could not be created

Status	Branch	Result
❌	9.0	Conflict resolution was aborted by the user
✅	8.18

Manual backport

To create the backport manually run:

backport --pr 124149

Questions ?

Please refer to the Backport tool documentation

When reindexing a data stream, the ILM metadata is copied from the index metadata of the source index to the destination index. But the ILM state of the new index can be stuck if the source index was in an AsyncAction at the time of reindexing. To un-stick the new index, we call TransportRetryAction to retry the AsyncAction. In the past this action would only run if the index were in the error phase. This change includes an update to TransportRetryAction, which allows it to be run when the index is not in an error phase, if the parameter requireError is set to false. (cherry picked from commit 10a8dcf) # Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java # x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/user/InternalUsers.java # x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/action/TransportRetryAction.java # x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/DataStreamsUpgradeIT.java

…124264) * Retry ILM async action after reindexing data stream (#124149) When reindexing a data stream, the ILM metadata is copied from the index metadata of the source index to the destination index. But the ILM state of the new index can be stuck if the source index was in an AsyncAction at the time of reindexing. To un-stick the new index, we call TransportRetryAction to retry the AsyncAction. In the past this action would only run if the index were in the error phase. This change includes an update to TransportRetryAction, which allows it to be run when the index is not in an error phase, if the parameter requireError is set to false. (cherry picked from commit 10a8dcf) # Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java # x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/user/InternalUsers.java # x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/action/TransportRetryAction.java # x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/DataStreamsUpgradeIT.java * Remove references to later transport versions * version for 8.19 was missing * index mode cannot be set on v7 indices

parkertimmins · 2025-03-06T22:38:09Z

💚 All backports created successfully

Status	Branch	Result
✅	9.0

Questions ?

Please refer to the Backport tool documentation

When reindexing a data stream, the ILM metadata is copied from the index metadata of the source index to the destination index. But the ILM state of the new index can be stuck if the source index was in an AsyncAction at the time of reindexing. To un-stick the new index, we call TransportRetryAction to retry the AsyncAction. In the past this action would only run if the index were in the error phase. This change includes an update to TransportRetryAction, which allows it to be run when the index is not in an error phase, if the parameter requireError is set to false. (cherry picked from commit 10a8dcf) # Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java # x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/action/TransportRetryAction.java

…124270) * Retry ILM async action after reindexing data stream (#124149) When reindexing a data stream, the ILM metadata is copied from the index metadata of the source index to the destination index. But the ILM state of the new index can be stuck if the source index was in an AsyncAction at the time of reindexing. To un-stick the new index, we call TransportRetryAction to retry the AsyncAction. In the past this action would only run if the index were in the error phase. This change includes an update to TransportRetryAction, which allows it to be run when the index is not in an error phase, if the parameter requireError is set to false. (cherry picked from commit 10a8dcf) # Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java # x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/user/InternalUsers.java # x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/action/TransportRetryAction.java # x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/DataStreamsUpgradeIT.java * index mode cannot be set on v7 indices

When reindexing a data stream, the ILM metadata is copied from the index metadata of the source index to the destination index. But the ILM state of the new index can be stuck if the source index was in an AsyncAction at the time of reindexing. To un-stick the new index, we call TransportRetryAction to retry the AsyncAction. In the past this action would only run if the index were in the error phase. This change includes an update to TransportRetryAction, which allows it to be run when the index is not in an error phase, if the parameter requireError is set to false. (cherry picked from commit 10a8dcf) # Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java # x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/action/TransportRetryAction.java

When reindexing a data stream, the ILM metadata is copied from the index metadata of the source index to the destination index. But the ILM state of the new index can be stuck if the source index was in an AsyncAction at the time of reindexing. To un-stick the new index, we call TransportRetryAction to retry the AsyncAction. In the past this action would only run if the index were in the error phase. This change includes an update to TransportRetryAction, which allows it to be run when the index is not in an error phase, if the parameter requireError is set to false.

Retry ILM async action after reindexing data stream

a1f0b89

elasticsearchmachine added the v9.1.0 label Mar 5, 2025

parkertimmins requested a review from masseyke March 5, 2025 20:37

parkertimmins added :StorageEngine/Data streams Data streams and their lifecycles v9.0.0 v8.18.0 v8.18.1 v8.19.0 auto-backport Automatically create backport pull requests when merged labels Mar 5, 2025

Do not test closed indices if ILM is enabled

a85b559

parkertimmins marked this pull request as ready for review March 5, 2025 20:44

elasticsearchmachine added the Team:Data Management (obsolete) DO NOT USE. This team no longer exists. label Mar 5, 2025

parkertimmins and others added 3 commits March 5, 2025 14:45

Merge branch 'main' into reindex-data-stream-async-action-ilm

b454405

transport version should use isPatch for previous versions

192d6cd

[CI] Auto commit changes from spotless

06e4bba

masseyke reviewed Mar 5, 2025

View reviewed changes

parkertimmins and others added 7 commits March 5, 2025 15:25

by default retry should require ilm error

3fba643

add comment

371ebb1

Skip async action if next step is empty

9ce5e22

[CI] Auto commit changes from spotless

bcfe075

variables in log in wrong order

0d3dedf

[CI] Auto commit changes from spotless

38b0570

Don't use time series index as complicates ILM tests

d3e680f

masseyke approved these changes Mar 6, 2025

View reviewed changes

masseyke requested a review from dakrone March 6, 2025 17:18

Merge branch 'main' into reindex-data-stream-async-action-ilm

130bf5b

parkertimmins added the >enhancement label Mar 6, 2025

Update docs/changelog/124149.yaml

db24d55

Missing semicolon!

7ad2d49

parkertimmins merged commit 10a8dcf into elastic:main Mar 6, 2025
17 checks passed

elasticsearchmachine added the backport pending label Mar 6, 2025

parkertimmins mentioned this pull request Mar 6, 2025

[8.x] Retry ILM async action after reindexing data stream (#124149) #124264

Merged

parkertimmins mentioned this pull request Mar 6, 2025

[8.18] Retry ILM async action after reindexing data stream (#124149) #124270

Merged

parkertimmins mentioned this pull request Mar 6, 2025

[9.0] Retry ILM async action after reindexing data stream (#124149) #124280

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retry ILM async action after reindexing data stream#124149

Retry ILM async action after reindexing data stream#124149
parkertimmins merged 15 commits intoelastic:mainfrom
parkertimmins:reindex-data-stream-async-action-ilm

parkertimmins commented Mar 5, 2025

elasticsearchmachine commented Mar 5, 2025

masseyke Mar 5, 2025

masseyke left a comment

elasticsearchmachine commented Mar 6, 2025

Uh oh!

elasticsearchmachine commented Mar 6, 2025

parkertimmins commented Mar 6, 2025

parkertimmins commented Mar 6, 2025

parkertimmins commented Mar 6, 2025

Labels

3 participants

Conversation

parkertimmins commented Mar 5, 2025

elasticsearchmachine commented Mar 5, 2025

masseyke Mar 5, 2025

Choose a reason for hiding this comment

masseyke left a comment

Choose a reason for hiding this comment

elasticsearchmachine commented Mar 6, 2025

Uh oh!

elasticsearchmachine commented Mar 6, 2025

💔 Backport failed

parkertimmins commented Mar 6, 2025

💔 Some backports could not be created

Manual backport

Questions ?

parkertimmins commented Mar 6, 2025

💔 Some backports could not be created

Manual backport

Questions ?

parkertimmins commented Mar 6, 2025

💚 All backports created successfully

Questions ?

Labels

3 participants