Hack to set write load to zero when balancing search nodes

[nicktindall]

Not for merging, just an attempt at a simple hack that should remove write load from search nodes when balancing.

It'd probably be better to be able to set the writeLoadBalance weight to zero for the WeightFunction used on search-only nodes, that way the node weight could go to zero on search nodes, but doing it this way should at least remove the benefit of any search-only shard movements purely for write load reasons. And it's a very simple POC.

It's not clear to me what effect this will have. The "average write load" should be reduced because the search nodes won't contribute to it, and the search node weights will probably remain persistently non-zero because there will always be the non-zero theta2 * (0 - avgWriteLoadPerNode) component.

Relates: ES-11367

[pxsalehi]

I was thinking of something like this initially when we discussed applying different weights, but doing it on the weight function (depending on the node role) and also change it for index-only nodes where we'd not consider the shard count. Obviously just as an experiment. This would be a simple change to use zero for shard count on the indexing tier, and write load on the search tier, w/o having to introduce new weights/settings. I think this is worth benchmarking and trying out. But I also don't know if it will lead to undesirable side-effects. AFAIU, while the current linear-weight-function-based balancing is inherently flawed, the separate weights on each tier would at least allow us to get the most of this approach, w/o affecting stateful.

[pxsalehi]

(ignoring shard count on the indexing tier would be meaningful if datastreams are used rather than normal indices, since for normal indices we don't have a write load)

[nicktindall]

but doing it on the weight function (depending on the node role)

Yes that seems like a better approach, being able to adjust the weights depending on the node role. I would have done that for the POC but it wasn't immediately apparent how to (still unfamiliar with the structure of this metadata)

[pxsalehi]

it wasn't immediately apparent how to

I think you can just add a helper to ModelNode that can check the RoutingNode's role.

[nicktindall]

Implemented properly in #126091