[Aggs] Force return 0 on empty buckets on count if null flag is disabled

[dej611]

Summary

Fixes #206555

This PR is an attempt to address the null bucket issue with count in Lens formula via the emptyAsNull flag.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

Risks

This PR introduces potentially some breaking changes, as count null values, in particular coming from shifted computations, as now converted to 0 if the flag has been enabled.
This change is not news in the code base as other aggs like distinct_count or value_count already implements it, but not count.
Apparently no test failed with this change, I've also added new unit ones to freeze the current behaviour and detect future changes.

[dej611]

/ci

[elasticmachine]

Pinging @elastic/kibana-visualizations (Team:Visualizations)

[lukasolson]

Code & functionality LGTM

[markov00]

This doesn't fully solve what was requested in the original issue.
null/zero is one thing, but the fact that count with timeshift doesn't use the extended_bounds is not addressed by the PR.
Actually not just count but also other metrics doesn't include the extended_bounds when using timeshift.
This results is missing buckets at the beginning and end of the table if they are empty/zero.

[markov00]

If I'm not wrong the current concept of emptyAsNull means: that if we find a bucket with a zero value, we will display it as null/missing.

 if (value === 0 && agg.params.emptyAsNull) {
        return null;
      }

But in this PR we are also doing the opposite: we are saying that null buckets can be shown as "empty" (actually zero) even if this wasn't requested in the issue and even not expressed by the parameter emptyAsNull.

[dej611]

This doesn't fully solve what was requested in the original issue. null/zero is one thing, but the fact that count with timeshift doesn't use the extended_bounds is not addressed by the PR. Actually not just count but also other metrics doesn't include the extended_bounds when using timeshift. This results is missing buckets at the beginning and end of the table if they are empty/zero.

I do not think the problem is related with the extended_bounds.
All the aggs can return null values when there's no data, with few exceptions which are count, distinct_count, values_count and sum, who expose a specific flag emptyAsNull to enforce this behaviour: now this null behaviour has a side effect within a Lens formula context which is to wipe out other results within a bucket. To resolve this the defaults Lens formula function has been introduced.
Now, back to the exceptions, the 4 functions above, in the scenario where count was using a time_shift was not honouring the opposite case: return 0 by default when the flag was not enabled.
This PR addresses this specific case, which will solve the issue with count and time_shift.

If we want to extends this behaviour to other aggs as well I think we'll need to rethink the agg model and how they work.

[dej611]

Discussed offline with @markov00 this PR.
The linked issue #206555 has a wider scope, which is about forcing ES to return all the buckets in any scenario when using a time shift.
That issue can lead to some wrong results in some cases OR partial results where values should be 0 in others.
This PR provides a solution for specific scenario where the count with shift is used together with another metric in a Lens formula, which resolves the "wrong results" part in the issue above. Not the partial result section tho.

For the partial result we've tested a PoC who seems to work, but needs to be validated in a follow up PR.

[ppisljar]

LGTM

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 69a7d58

Failed CI Steps

• FTR Configs #113

Test Failures

• [job] [logs] FTR Configs #113 / Rules Management - Rule Bulk Action API @ess perform_bulk_action - ESS specific logic should disable rules and migrate actions

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
data	423.6KB	423.6KB	+36.0B

History

• 💔 Build #271967 failed 35a47f1
• 💔 Build #271928 failed e66661d
• 💚 Build #270379 succeeded 51546d5
• 💛 Build #268914 was flaky 37d3001
• 💚 Build #268712 succeeded ef67968

[kibanamachine]

Starting backport for target branches: 9.0

https://github.com/elastic/kibana/actions/runs/13155950588

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	9.0

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation