Skip to content

TransportGetAllocationStatsAction may cause significant load on elected master #110716

New issue
New issue
Closed
#124898
Closed
TransportGetAllocationStatsAction may cause significant load on elected master#110716
#124898
Labels
:Distributed/DistributedA catch all label for anything in the Distributed Area. Please avoid if you can.>bugTeam:DistributedMeta label for distributed team.
@idegtiarenko

Description

@idegtiarenko
idegtiarenko
opened on Jul 10, 2024

Elasticsearch Version

8.14

Installed Plugins

No response

Java Version

bundled

OS Version

any

Problem Description

TransportGetAllocationStatsAction runs on elected master so it is theoretically possible to overload it by executing node stats requests around various nodes in cluster, especially in a clusters with many shards as complexity is proportional to the shard count.

The aggregated result of the computation is small (5 numbers per node), we should consider caching it for small period of time (1 minute?) and reuse it between different calls during.

elasticsearch/server/src/main/java/org/elasticsearch/action/admin/cluster/node/stats/TransportNodesStatsAction.java

Lines 88 to 96 in 5e52059

if (NodesStatsRequestParameters.Metric.ALLOCATIONS.containedIn(metrics)) {
client.execute(
TransportGetAllocationStatsAction.TYPE,
new TransportGetAllocationStatsAction.Request(new TaskId(clusterService.localNode().getId(), task.getId())),
listener.delegateFailure((l, r) -> {
ActionListener.respondAndRelease(l, newResponse(request, merge(responses, r.getNodeAllocationStats()), failures));
})
);
} else {

Steps to Reproduce

n/a

Logs (if relevant)

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    :Distributed/DistributedA catch all label for anything in the Distributed Area. Please avoid if you can.>bugTeam:DistributedMeta label for distributed team.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions