Disable Real Memory Circuit Breaker by default and deprecate it

[original-brownbear]

As discussed, deprecating the setting and flipping it to false by default on account of the progress we made that makes this breaker hurt more than it helps.

[github-actions]

Documentation preview:

• ✨ Changed pages

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[iverase]

I understand the motivation of this change and I agree with it. Still I am concerned that some parts of the code rely on the real memory circuit breaker in order to avoid OOMs. This change will leave those cases totally out of coverage so I think we need to provide some other mechanism in order to have some protection against OOMs.

The cases I am aware of:

elasticsearch/server/src/main/java/org/elasticsearch/search/aggregations/MultiBucketConsumerService.java

Line 133 in e1a091f

breaker.addEstimateBytesAndMaybeBreak(0, "allocated_buckets");

This is pretty important and the reason we increase the max number of buckets from 10k to 65k as it will give the protection in case we will be crashing too much data on heap on the coordinator node during reduction.

elasticsearch/server/src/main/java/org/elasticsearch/search/aggregations/bucket/BucketsAggregator.java

Line 87 in 0da5220

breaker.addEstimateBytesAndMaybeBreak(0, "allocated_buckets");

Similar to the above but in the data node. There are some objects in the aggregation framework that are not memory tracked.

elasticsearch/server/src/main/java/org/elasticsearch/index/fielddata/ordinals/GlobalOrdinalsBuilder.java

Line 65 in 5516579

breaker.addEstimateBytesAndMaybeBreak(0L, "Global Ordinals");

That was added recently when we notice that building global ordinals can hit OOMs in clusters with small heap.

elasticsearch/server/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/FiltersAggregator.java

Line 307 in 484bde9

addRequestCircuitBreakerBytes(0L);

That was added recently when we notice that building the filters on a filters aggregator can hit OOMs in clusters with small heap.