Skip to content

GET /_migration/deprecations doesn't report node deprecations if low watermark exceeded #137004

@DaveCTurner

Description

@DaveCTurner

This seemingly-innocuous code ...

... doesn't work because issues came from Stream#toList and is therefore immutable. Instead it throws a UOE:

java.lang.UnsupportedOperationException: null
        at java.util.ImmutableCollections.uoe(ImmutableCollections.java:159) ~[?:?]
        at java.util.ImmutableCollections$AbstractImmutableCollection.add(ImmutableCollections.java:164) ~[?:?]
        at org.elasticsearch.xpack.deprecation.TransportNodeDeprecationCheckAction.nodeOperation(TransportNodeDeprecationCheckAction.java:146) ~[?:?]
        at org.elasticsearch.xpack.deprecation.TransportNodeDeprecationCheckAction.nodeOperation(TransportNodeDeprecationCheckAction.java:110) ~[?:?]
        at org.elasticsearch.xpack.deprecation.TransportNodeDeprecationCheckAction.nodeOperation(TransportNodeDeprecationCheckAction.java:43) ~[?:?]
        at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperationAsync(TransportNodesAction.java:261) ~[elasticsearch-8.19.5.jar:?]
        at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.lambda$messageReceived$0(TransportNodesAction.java:277) ~[elasticsearch-8.19.5.jar:?]
        at org.elasticsearch.action.ActionListener.run(ActionListener.java:454) ~[elasticsearch-8.19.5.jar:?]
        at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:275) ~[elasticsearch-8.19.5.jar:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:86) ~[elasticsearch-8.19.5.jar:?]
        at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:1078) ~[elasticsearch-8.19.5.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-8.19.5.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.19.5.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[?:?]
        at java.lang.Thread.run(Thread.java:1474) ~[?:?]

Such node-level failures do not propagate back to the API response, they're just logged and then silently dropped here:

logger.warn("nodes failed to run deprecation checks: {}", failedNodeIds);
for (FailedNodeException failure : response.failures()) {
logger.debug("node {} failed to run deprecation checks: {}", failure.nodeId(), failure);
}

This sort of node-level failure is more likely than you might think because we report low watermark violations far too liberally.

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions