[ES|QL] Making the Expression Suggestor More Semantically Intelligent by bartoval · Pull Request #241081 · elastic/kibana

bartoval · 2025-10-29T09:08:06Z

Summary

This PR has a multiple purpose:

Improve the quality of context-based suggestions, especially for multiple, conditional, and homogeneous signatures. (CASE, COALESCE, BUCKETS)
prediction of upcoming fields based on signature combinations (already present but improved)
Resolve old ambiguities present in old versions that cause errors in queries (such as suggesting Integer fields when requesting constant integers)
Strengthen the suggest-for-expression mechanism.
Improve the quality of suggestion tests.

Important notes:

We've currently limited TEXT/KEYWORDS suggestions (only inside functions) to operators like is (not) null, like, etc., removing comparison operators. This point need a bit clarification because theoretically it does make sense suggest comparison operators.
- STATS doesn't have a location for those operators either, so for example STATS COUNT_DISTINCT(agent only shows the comma and not operators like is (not) null
We now distinguish between integers (long, double) and constants, which means that in some cases we don't suggest anything. For example, ROUNT_TO has an optional precision field that must be a numeric constant. Previously, we also suggested numeric fields (like bytes), which made the Elastic query invalid. This case can work well with PR [ES|QL] show function signature while typing #180528

some examples

**Only Constant and variadic tests **
FROM kibana_sample_data_logs | EVAL ROUND_TO(bytes, // we don't suggest nothing because we expect numeric constants
FROM kibana_sample_data_logs | EVAL ROUND_TO(bytes, 3 // we know that this function has max 2 parameters and we suggest only math opearators, because they are the only ones that make sense
FROM kibana_sample_data_logs | EVAL CONCAT(agent, agent.keywords // CONCAT has min 2 params but it can continue, so we suggest a comma

Homogenity tests
FROM kibana_sample_data_logs | EVAL COALESCE(bytes // Here we suggest numeric, comma and comparison operators (because we can transform it into boolean

FROM kibana_sample_data_logs | EVAL COALESCE(bytes, // After the comma we know that the subsequent types must be numeric and we suggest only these (and subsequently the operators that make sense)

Conditional cases
FROM kibana_sample_data_logs | EVAL CASE(bytes // The first parameter must be a complete Boolean expression, so I don't have to suggest a comma. I suggest a comma only when the right operand also exists.

Multisignatures
FROM kibana_sample_data_logs | STATS COUNT_DISTINCT(agent) BY BUCKETS(bytes, // It's a number and I only expect a second constant parameter, so I don't suggest anything

FROM kibana_sample_data_logs | STATS COUNT_DISTINCT(agent) BY BUCKETS(timestamp, // Here I suggest 1 month, 1 year... because it could be a date or a numeric constant (no suggestion). If I select 1 month, then I don't suggest anything because the signature says I have a maximum of 2 parameters.

FROM kibana_sample_data_logs | STATS COUNT_DISTINCT(agent) BY BUCKETS(timestamp, 3, // Here the second parameter is a constant, so the rules change. I unlock parameters 3 and 4 with the possible (constant) values: ?start . ?end, select from datepicker

elasticmachine · 2025-10-29T09:09:11Z

Pinging @elastic/kibana-esql (Team:ESQL)

bartoval · 2025-10-29T09:21:03Z

One of the purposes of this PR is also to manually test to see if something has been forgotten or not totally correct

stratoula

Looks great and I could not find any bug, only better experience ❤️

I just have some questions that I would like you to answer first

stratoula · 2025-10-29T09:11:51Z

src/platform/packages/shared/kbn-esql-ast/src/__tests__/autocomplete.ts

 };
+
+// Re-export operator groups for use in tests (avoid hardcoding operator names)
+export {


Why are you exporting here again? We have them already here ../definitions/all_operators. Why dont you use the existings exports for the tests?

Thanks,
At the beginning it was hardcoded and I had all the references to this structure, then I changed it but I forgot to delete it and update

stratoula · 2025-10-29T10:00:26Z

...atform/packages/shared/kbn-esql-ast/src/commands_registry/commands/eval/autocomplete.test.ts

-      ],
-      mockCallbacks
-    );
+    await evalExpectSuggestions('from a | eval a=round(doubleField, ', [], mockCallbacks);


stratoula · 2025-10-29T11:17:35Z

...atform/packages/shared/kbn-esql-ast/src/commands_registry/commands/eval/autocomplete.test.ts


    test('suggests operators after initial column based on type', async () => {
      // case( field ) suggests all appropriate operators for that field type
+      // Note: CASE is expression-heavy, comma is not automatically suggested after fields


It also has this bug (but it happens in main so it is not because of this PR)

We should open an issue to track it

stratoula · 2025-10-29T11:53:22Z

...tform/packages/shared/kbn-esql-ast/src/commands_registry/commands/stats/autocomplete.test.ts

          expect(suggestions).toContainEqual(expectedCompletionItem);
        });
+
+        test('should NOT appear inside BUCKET function', async () => {


It took me some time to understand. This is a suggestion, change it if you dont like it but we need to adjust it

Suggested change

test('should NOT appear inside BUCKET function', async () => {

test('BUCKET constant arguments should not trigger function suggestions', async () => {

stratoula · 2025-10-29T11:54:01Z

...tform/packages/shared/kbn-esql-ast/src/commands_registry/commands/stats/autocomplete.test.ts

+        expect(duplicates).toEqual([]);
+      });
+
+      test('BUCKET with numeric field should NOT show col0 or date histogram at second param', async () => {


stratoula · 2025-10-29T11:59:42Z

...ackages/shared/kbn-esql-validation-autocomplete/src/autocomplete/__tests__/functions.test.ts

    });

-    it('conditional with text field: suggests comparison/pattern/IN/null operators in STATS', async () => {
+    it.skip('conditional with text field: suggests pattern/IN/null operators (not comparison) in STATS', async () => {


Why do you skip this poor test?

Because he failed, I'm touchy and I punished him. :D

Now I have forgiven him

sddonne

Looks amazing! these are really nice improvements ❤️

sddonne · 2025-10-29T12:41:45Z

src/platform/packages/shared/kbn-esql-ast/src/commands_registry/commands/stats/autocomplete.ts

FROM kibana_sample_data_logs | STATS COUNT_DISTINCT(agent) BY BUCKETS(timestamp, 3, // Here the second parameter is a constant, so the rules change. I unlock parameters 3 and 4 with the possible (constant) values: ?start . ?end, select from datepicker

Not sure if it's simple enough to achieve, but would be nice that BUCKET suggests date functions for the last 2 args. To be able to write querys like this:

FROM ebt-kibana-browser | STATS BY BUCKET(context.cloudTrialEndDate, 30, "2012-10-15T11:27:38.230Z", NOW())

do you mean when you press Add date histogram?

ah ok you are referring to NOW()

stratoula

Thanx Val!

elasticmachine · 2025-10-29T21:22:23Z

💔 Build Failed

Buildkite Build
Commit: f9359e4

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #71 / discover/tabs discover - ES|QL controls should add an ES|QL multi - value control
[job] [logs] FTR Configs #71 / discover/tabs discover - ES|QL controls should add an ES|QL multi - value control
[job] [logs] Scout: [ platform / discover_enhanced ] plugin / stateful - Discover app - saved searches - should unselect saved search when navigating to a 'new'

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`esql`	578.3KB	578.8KB	+480.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`kbnUiSharedDeps-srcJs`	4.0MB	4.0MB	+2.9KB

History

💔 Build #356387 failed b09f794
💛 Build #356204 was flaky 5a89150
💚 Build #356087 succeeded c7916ef

cc @bartoval

…elastic#241081) ## Summary elastic#239507 This PR has a multiple purpose: - Improve the quality of context-based suggestions, especially for multiple, conditional, and homogeneous signatures. (CASE, COALESCE, BUCKETS) - prediction of upcoming fields based on signature combinations (already present but improved) - Resolve old ambiguities present in old versions that cause errors in queries (such as suggesting Integer fields when requesting constant integers) - Strengthen the suggest-for-expression mechanism. - Improve the quality of suggestion tests. **Important notes:** - We've currently limited TEXT/KEYWORDS suggestions (only inside functions) to operators like is (not) null, like, etc., removing comparison operators. This point need a bit clarification because theoretically it does make sense suggest comparison operators. - **STATS** doesn't have a location for those operators either, so for example` STATS COUNT_DISTINCT(agent` only shows the comma and not operators like is (not) null - We now distinguish between integers (long, double) and constants, which means that in some cases **we don't suggest** anything. For example, ROUNT_TO has an optional precision field that must be a numeric constant. Previously, we also suggested numeric fields (like bytes), which made the Elastic query invalid. This case can work well with PR elastic#180528 **some examples** **Only Constant and variadic tests ** `FROM kibana_sample_data_logs | EVAL ROUND_TO(bytes,` // we don't suggest nothing because we expect numeric constants `FROM kibana_sample_data_logs | EVAL ROUND_TO(bytes, 3` // we know that this function has max 2 parameters and we suggest only math opearators, because they are the only ones that make sense `FROM kibana_sample_data_logs | EVAL CONCAT(agent, agent.keywords` // CONCAT has min 2 params but it can continue, so we suggest a comma **Homogenity tests** `FROM kibana_sample_data_logs | EVAL COALESCE(bytes` // Here we suggest numeric, comma and comparison operators (because we can transform it into boolean `FROM kibana_sample_data_logs | EVAL COALESCE(bytes,` // After the comma we know that the subsequent types must be numeric and we suggest only these (and subsequently the operators that make sense) **Conditional cases** `FROM kibana_sample_data_logs | EVAL CASE(bytes` // The first parameter must be a complete Boolean expression, so I don't have to suggest a comma. I suggest a comma only when the right operand also exists. **Multisignatures** `FROM kibana_sample_data_logs | STATS COUNT_DISTINCT(agent) BY BUCKETS(bytes,` // It's a number and I only expect a second constant parameter, so I don't suggest anything `FROM kibana_sample_data_logs | STATS COUNT_DISTINCT(agent) BY BUCKETS(timestamp,` // Here I suggest 1 month, 1 year... because it could be a date or a numeric constant (no suggestion). If I select 1 month, then I don't suggest anything because the signature says I have a maximum of 2 parameters. `FROM kibana_sample_data_logs | STATS COUNT_DISTINCT(agent) BY BUCKETS(timestamp, 3, ` // Here the second parameter is a constant, so the rules change. I unlock parameters 3 and 4 with the possible (constant) values: ?start . ?end, select from datepicker

bartoval added 3 commits October 28, 2025 19:26

improve suggest for expression policies

6cc83a3

filter precision as constantonly

7c1e048

improve roles

c7916ef

bartoval requested a review from a team as a code owner October 29, 2025 09:08

bartoval self-assigned this Oct 29, 2025

bartoval added Feature:ES|QL ES|QL related features in Kibana Team:ESQL ES|QL related features in Kibana t// labels Oct 29, 2025

bartoval added backport:skip This PR does not require backporting release_note:enhancement v9.3.0 labels Oct 29, 2025

stratoula reviewed Oct 29, 2025

View reviewed changes

sddonne approved these changes Oct 29, 2025

View reviewed changes

stratoula approved these changes Oct 29, 2025

View reviewed changes

update from feedbacks

cb6092c

bartoval force-pushed the improve_suggestions branch from 5a89150 to cb6092c Compare October 29, 2025 17:29

Merge branch 'main' into improve_suggestions

b09f794

Merge branch 'main' into improve_suggestions

9d9cebb

bartoval force-pushed the improve_suggestions branch from f9359e4 to 9d9cebb Compare October 29, 2025 22:17

bartoval merged commit 7582d50 into elastic:main Oct 30, 2025
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ES|QL] Making the Expression Suggestor More Semantically Intelligent#241081

[ES|QL] Making the Expression Suggestor More Semantically Intelligent#241081
bartoval merged 6 commits intoelastic:mainfrom
bartoval:improve_suggestions

bartoval commented Oct 29, 2025 •

edited

Loading

elasticmachine commented Oct 29, 2025

bartoval commented Oct 29, 2025

stratoula left a comment

stratoula Oct 29, 2025 •

edited

Loading

bartoval Oct 29, 2025

stratoula Oct 29, 2025

stratoula Oct 29, 2025

stratoula Oct 29, 2025

stratoula Oct 29, 2025

stratoula Oct 29, 2025

bartoval Oct 29, 2025

sddonne left a comment

sddonne Oct 29, 2025

bartoval Oct 29, 2025

bartoval Oct 29, 2025

stratoula left a comment

elasticmachine commented Oct 29, 2025 •

edited

Loading

Uh oh!

Labels

4 participants

	test('should NOT appear inside BUCKET function', async () => {
	test('BUCKET constant arguments should not trigger function suggestions', async () => {

Conversation

bartoval commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

elasticmachine commented Oct 29, 2025

bartoval commented Oct 29, 2025

stratoula left a comment

Choose a reason for hiding this comment

stratoula Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sddonne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stratoula left a comment

Choose a reason for hiding this comment

elasticmachine commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

Page load bundle

History

Uh oh!

Labels

4 participants

bartoval commented Oct 29, 2025 •

edited

Loading

stratoula Oct 29, 2025 •

edited

Loading

elasticmachine commented Oct 29, 2025 •

edited

Loading