Improve accuracy of write load forecast when shard numbers change

[nicktindall]

When the number of shards changes in a data-stream, whether due to auto-sharding or manual intervention, it impacts the write-load forecast used for the new write index. This is because the forecast is calculated as a weighted average per-shard write load. So if the number of shards increased in the new write index, the indexes total write load will increase correspondingly, likewise if the number of shards decreased. This PR shifts to calculating a per-index write load, so it can be correctly adjusted when the shard count changes.

Explanation

Write load is calculated per shard, it is equal to
WRITE_LOAD(shard) = WRITE_TIME(shard) / UPTIME(shard)

To calculate the forecasted write load of the shards of a new index on DS rollover, we calculate the weighted average write load of all the shards of all the "recent" indices, e.g.
FORECAST_WRITE_LOAD(shard) = sum(WRITE_LOAD(shard)) / sum(UPTIME(shard)) | all shard in RECENT_INDEX

We use the forecasted write load quite heavily in balancing.

To calculate the write load for a node, we add the write loads of all the shards in all the allocated indices
FORECAST_WRITE_LOAD(node) = sum(FORECAST_WRITE_LOAD(shard)) | all shards on node
(see org.elasticsearch.cluster.routing.allocation.allocator.BalancedShardsAllocator.ModelNode#(addShard|removeShard))

And to calculate the write load for an index, we add the write loads of all the shards in the index
FORECAST_WRITE_LOAD(index) = sum(FORECAST_WRITE_LOAD(shard)) | all shards in index (taking into account replicas)
(see org.elasticsearch.cluster.routing.allocation.allocator.WeightFunction#getIndexWriteLoad)

This means when auto-sharding decides to increase the number of shards, the total write load in the cluster increases, because
e.g. an index that went from 3 shards to 6 shards goes from 3 * FORECAST_WRITE_LOAD(shard) to 6 * FORECAST_WRITE_LOAD(shard)
Conversely when auto-sharding decides to decrease the number of shards, the total write load in the cluster will decrease

Why is this a problem?

If we auto-shard down from 63 to 38 shards in a rollover, like we did the scale testing environment, the balancer will use the forecasted write-load that was calculated for 63 shards, but with only 38 shards. That load from 63 shards will be compressed into the 38 shards that replace them. So the balancer will under-estimate the write load for each shard by (1 - (38/36)) = ~40%. This will make it think it's in a desired state, but will leave nodes that have a greater proportion of the scaled-down (hence under-estimated) index's shards overloaded.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[nicktindall]

Fixed a typo too

[elasticsearchmachine]

Hi @nicktindall, I've created a changelog YAML for you.

[mhl-b]

I found testWriteLoadForecast covers too many cases, hard to follow which test validates your change. Can you add tests for these two cases when number of shards goes up and down and how it impacts forecast, please?

[nicktindall]

There are sometimes small rounding errors

[mhl-b]

LGTM