More efficient sort in tryRelocateShard#128063
Merged
elasticsearchmachine merged 3 commits intoelastic:mainfrom May 19, 2025
Merged
More efficient sort in tryRelocateShard#128063elasticsearchmachine merged 3 commits intoelastic:mainfrom
tryRelocateShard#128063elasticsearchmachine merged 3 commits intoelastic:mainfrom
Conversation
No need to do this via an allocation-heavy `Stream`, we can just put the objects straight into an array, sort them in-place, and keep hold of the array to avoid having to allocate anything on the next iteration. Also slims down `BY_DESCENDING_SHARD_ID`: it's always sorting the same index so we don't need to look at `ShardId#index` in the comparison, nor do we really need multiple layers of vtable lookups, we can just compare the shard IDs directly. Relates elastic#128021
DaveCTurner
added
>enhancement
:Distributed/Allocation
Collaborator
|
Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination) |
elasticsearchmachine
added
the
Team:Distributed Coordination (obsolete)
Collaborator
|
Hi @DaveCTurner, I've created a changelog YAML for you. |
DaveCTurner
added
the
auto-backport
DiannaHohensee
approved these changes
May 19, 2025
Contributor
DiannaHohensee
left a comment
DiannaHohensee
left a comment
There was a problem hiding this comment.
lgtm 👍
To confirm, this change essentially
- removes the use of Stream, which isn't performant
- shortens the comparison method
- reuses allocated memory
Though this last one sacrifices memory in favor of performance -- if a user has one index with a stupid number of shards, we're keep that array memory allocation for the duration of a balancing round? Though even a large number of shards will take insignificant memory.
| for (final var shardRouting : index) { | ||
| if (shardRouting.started()) { // cannot rebalance unassigned, initializing or relocating shards anyway | ||
| shardRoutingsOnMaxWeightNode[startedShards] = shardRouting; | ||
| startedShards += 1; |
Contributor
There was a problem hiding this comment.
opt: ++startedShards or shardRoutingsOnMaxWeightNode[startedShards++] = shardRouting;
Contributor
Author
There was a problem hiding this comment.
IMO unary increments are harder on the reader than spelling out a += 1 so I'd rather leave it like this.
DaveCTurner
added
the
auto-merge-without-approval
Contributor
Author
Yeah the maximum is 1024, if they all end up on one node for some reason then we allocate an array of length 2048 which is still tiny and probably not worth GC'ing. We probably allocate more than 2048*8=16kiB today just by using streams. |
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this pull request
May 19, 2025
No need to do this via an allocation-heavy `Stream`, we can just put the objects straight into an array, sort them in-place, and keep hold of the array to avoid having to allocate anything on the next iteration. Also slims down `BY_DESCENDING_SHARD_ID`: it's always sorting the same index so we don't need to look at `ShardId#index` in the comparison, nor do we really need multiple layers of vtable lookups, we can just compare the shard IDs directly. Relates elastic#128021
Collaborator
💚 Backport successful
|
elasticsearchmachine
pushed a commit
that referenced
this pull request
May 19, 2025
No need to do this via an allocation-heavy `Stream`, we can just put the objects straight into an array, sort them in-place, and keep hold of the array to avoid having to allocate anything on the next iteration. Also slims down `BY_DESCENDING_SHARD_ID`: it's always sorting the same index so we don't need to look at `ShardId#index` in the comparison, nor do we really need multiple layers of vtable lookups, we can just compare the shard IDs directly. Relates #128021
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No need to do this via an allocation-heavy
Stream, we can just put theobjects straight into an array, sort them in-place, and keep hold of the
array to avoid having to allocate anything on the next iteration.
Also slims down
BY_DESCENDING_SHARD_ID: it's always sorting the sameindex so we don't need to look at
ShardId#indexin the comparison, nordo we really need multiple layers of vtable lookups, we can just compare
the shard IDs directly.
Relates #128021