[ES|QL Controls] Refresh "Values from a query" options on dashboard reload by Zacqary · Pull Request #225101 · elastic/kibana

Zacqary · 2025-06-24T15:09:00Z

Summary

For "values from a query" ES|QL controls, attempts to execute the query again on dashboard page load, refresh, timerange change, etc. to get an up-to-date list of values.

If the query execution fails, values cached at the previous edit time will be used.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

Zacqary · 2025-06-24T16:41:35Z

src/platform/packages/shared/presentation/presentation_controls/esql/esql_query_to_options.ts

+  }
+};
+
+esqlQueryToOptions.isSuccess = (result: unknown): result is ESQLControlOptionsSuccess =>


I think glomming a type guard for the result onto the function itself is an elegant way to save space and imports but if this is too galaxy brain javascript to be a sustainable pattern, I'm open to changing this.

…ocs'

elasticmachine · 2025-06-24T22:28:00Z

Pinging @elastic/kibana-presentation (Team:Presentation)

nreese · 2025-06-25T00:32:36Z

src/platform/plugins/shared/controls/public/controls/esql_control/esql_control_selections.ts

+  ESQLVariableType,
+} from '@kbn/esql-types';
 import { PublishingSubject, StateComparators } from '@kbn/presentation-publishing';
+import { esqlQueryToOptions } from '@kbn/presentation-controls';


Why is this placed in a package? What are the future plans for this package and method?

We need to make this method available to the esql plugin and controls plugin without creating any new cross-dependencies, because in my experience that eventually leads to circular dependency hell. I wasn't able to find an existing kbn package that seemed to fit.

If we're going to work on combining the UI components for classic and ES|QL controls, I expect we're going to run into more cases like this.

Thanks for the explanation. That makes sense.

Why does it need its own package and is that it important that it it has presentation-controls namespace? Would it be better to place in an existing esql package like esql-utils? That way, all esql logic is located in a single location.

Sure, I can move it to esql-utils and hold off on creating a new package for controls until we've got a larger use case.

I am not sure we need to move it. I only want to have the discussion and understand the why. If there are no plans that would require a separate package and no design reasons, then we can have the conversation about moving it into an existing package.

stratoula · 2025-06-25T10:04:30Z

I think this will create conflicts with #225054 but it should be easy to solve, let me know if you need any help

stratoula

I didnt go through the code but I would like to raise a concern about this feature mostly due to how ES|QL works and returns results

Let'say that the user types a query like this:

FROM logs* | STATS BY agent.name | LIMIT 5

This returns 5 values (one is null and we dont allow it)

Now every time you refresh your dashboard this list changes

and can change so much that the selected value is not part of the list anymore

which can be weird. This happens due to the randomness of the data retrieval especially for datasets with sparse data.

A user can improve that with sorting etc but it something I wanted to raise with you before we move on with this functionality cc @ghudgins

...form/plugins/shared/esql/public/triggers/esql_controls/control_flyout/value_control_form.tsx

# Conflicts: # src/platform/plugins/shared/esql/public/triggers/esql_controls/control_flyout/value_control_form.tsx

nreese · 2025-06-27T18:33:38Z

src/platform/plugins/shared/controls/public/controls/esql_control/esql_control_selections.ts

+      availableOptions$.next(result.values);
+    }
+  }
+  const fetchSubscription = controlFetch$.subscribe(updateAvailableOptions);


This introduces a race condition. If updateAvailableOptions does not return before controlFetch$ fires again and updateAvailableOptions from the second emit returns first, then updateAvailableOptions from the first emit will overwrite the options. to solve this, you should use a switchMap

const fetchSubscription = controlFetch$.pipe( filter(() => controlType$.getValue() === EsqlControlType.VALUES_FROM_QUERY) switchMap(async (fetchContext) => { return await getESQLSingleColumnValues({ query: esqlQuery$.getValue(), search: dataService.search.search, timeRange: fetchContext.timeRange, }); }), ).subscribe(results => { if (getESQLSingleColumnValues.isSuccess(results)) { availableOptions$.next(result.values); } });

nreese · 2025-06-27T18:36:31Z

src/platform/plugins/shared/controls/public/controls/esql_control/esql_control_selections.ts

+export function initializeESQLControlSelections(
+  initialState: ESQLControlState,
+  controlFetch$: ReturnType<ControlGroupApi['controlFetch$']>,
+  timeRange$?: PublishingSubject<TimeRange | undefined>


You don't need to pass in timeRange$, its provided by controlFetch$

Zacqary · 2025-06-27T18:52:10Z

@stratoula Had a conversation with @ThomThomson and we've decided just to apply these changes to the controls plugin for now since we need to get this PR merged urgently. This duplicates some of the functionality the editor flyout already uses, but we'll resolve that in a follow-up.

nreese · 2025-06-27T22:30:23Z

.../plugins/shared/controls/public/controls/esql_control/utils/get_esql_single_column_values.ts

+      return { values };
+    }
+
+    return { columns, errors: [] };


I am not sure I understand why the function would return columns when there are more then one? Would this be an error case? What consumer would use this response? Would it be better to just throw an error if multiple columns are returned?

The flyout UI needed this because it needs to populate a dropdown list of possible columns in the error state. I can remove it for now.

I'm anticipating we'll need this again in a followup PR so I'm going to keep the values/errors key value schema in the return type

nreese

kibana-presentation changes LGTM after fixing last few comments in test. Thanks for cleaning everything up
code review only

nreese · 2025-06-30T14:35:42Z

...ins/shared/controls/public/controls/esql_control/utils/get_esql_single_column_values.test.ts

+      search: searchMock,
+    })) as GetESQLSingleColumnValuesSuccess;
+    expect(getESQLSingleColumnValues.isSuccess(result)).toBe(true);
+    expect('columns' in result).toBe(false);


columns is no longer in response so this expect is not needed.

nreese · 2025-06-30T14:36:32Z

...ins/shared/controls/public/controls/esql_control/utils/get_esql_single_column_values.test.ts

+      }
+    `);
+  });
+  it('returns columns and empty error on successful query that returns multiple columns', async () => {


how about renaming to 'returns error when query returns multiple columns'?

kibanamachine · 2025-06-30T17:20:40Z

Starting backport for target branches: 8.17, 8.18, 8.19, 9.0, 9.1

https://github.com/elastic/kibana/actions/runs/15979438288

elasticmachine · 2025-06-30T17:21:09Z

💚 Build Succeeded

Buildkite Build
Commit: d97cb71

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`controls`	357	384	+27

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`controls`	473.1KB	477.7KB	+4.6KB

History

💔 Build #313671 failed 8378974
💔 Build #313602 failed 583c491
💔 Build #313569 failed efb01f3
💚 Build #313303 succeeded 950b0ad
💔 Build #313283 failed ced7251
💔 Build #313260 failed 279735d

…eload (elastic#225101) ## Summary Closes elastic#222198 For "values from a query" ES|QL controls, attempts to execute the query again on dashboard page load, refresh, timerange change, etc. to get an up-to-date list of values. If the query execution fails, values cached at the previous edit time will be used. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 215a4bf)

kibanamachine · 2025-06-30T17:27:47Z

💔 Some backports could not be created

Status	Branch	Result
❌	8.17	Backport failed because of merge conflicts
❌	8.18	Backport failed because of merge conflicts
❌	8.19	Backport failed because of merge conflicts
❌	9.0	Backport failed because of merge conflicts
✅	9.1

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 225101

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2025-06-30T18:06:56Z

Starting backport for target branches: 8.19, 9.1

https://github.com/elastic/kibana/actions/runs/15980325200

kibanamachine · 2025-06-30T18:06:57Z

Starting backport for target branches: 8.19, 9.1

https://github.com/elastic/kibana/actions/runs/15980325283

kibanamachine · 2025-06-30T18:06:58Z

Starting backport for target branches: 8.19, 9.1

https://github.com/elastic/kibana/actions/runs/15980325244

Zacqary · 2025-06-30T18:30:03Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…eload (elastic#225101) ## Summary Closes elastic#222198 For "values from a query" ES|QL controls, attempts to execute the query again on dashboard page load, refresh, timerange change, etc. to get an up-to-date list of values. If the query execution fails, values cached at the previous edit time will be used. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 215a4bf) # Conflicts: # src/platform/plugins/shared/controls/tsconfig.json

…oard reload (#225101) (#225891) # Backport This will backport the following commits from `main` to `9.1`: - [[ES|QL Controls] Refresh "Values from a query" options on dashboard reload (#225101)](#225101)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Zac Xeper <Zacqary@users.noreply.github.com>

…board reload (#225101) (#225903) # Backport This will backport the following commits from `main` to `8.19`: - [[ES|QL Controls] Refresh "Values from a query" options on dashboard reload (#225101)](#225101)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

Zacqary added 3 commits June 23, 2025 16:22

Extract esqlQueryToOptions, run on each control init

8cc598a

Update ESQL options when use clicks refresh

7770834

Switch reload$ subscription to controlFetch$

8248892

Fix README, add typed results to esqlQueryToOptions

6bef22f

Zacqary commented Jun 24, 2025

View reviewed changes

Zacqary and others added 10 commits June 24, 2025 11:51

Remove console.trace

f482804

Update choose column button text to make more prominent

e9b9ad4

Fix values control form test

9b9cf7c

Add test for values from query update

4dccf60

Merge remote-tracking branch 'upstream/main' into 222198-esql-control-rt

4c8cced

[CI] Auto-commit changed files from 'node scripts/build_plugin_list_d…

02a14b4

…ocs'

Fix presentation-controls jest config

94d744f

Add test for esqlQueryToOptions

f382b4e

[CI] Auto-commit changed files from 'node scripts/notice'

bea79e7

Fix tsconfig

29bcd37

Zacqary marked this pull request as ready for review June 24, 2025 22:27

Zacqary requested review from a team as code owners June 24, 2025 22:27

nreese reviewed Jun 25, 2025

View reviewed changes

stratoula reviewed Jun 25, 2025

View reviewed changes

...form/plugins/shared/esql/public/triggers/esql_controls/control_flyout/value_control_form.tsx Outdated Show resolved Hide resolved

Zacqary and others added 2 commits June 25, 2025 10:56

Merge remote-tracking branch 'upstream/main' into 222198-esql-control-rt

3dcb2ec

# Conflicts: # src/platform/plugins/shared/esql/public/triggers/esql_controls/control_flyout/value_control_form.tsx

[CI] Auto-commit changed files from 'node scripts/yarn_deduplicate'

62e6dff

[CI] Auto-commit changed files from 'node scripts/notice'

68c786c

Zacqary enabled auto-merge (squash) June 27, 2025 18:29

nreese reviewed Jun 27, 2025

View reviewed changes

Fix race condition in inializeESQLControlSelections

8378974

Fix jest

3f9075a

nreese reviewed Jun 27, 2025

View reviewed changes

Zacqary added 2 commits June 30, 2025 09:33

Remove columns from getESQLSingleColumnValues return type

89b05d8

Merge remote-tracking branch 'upstream/main' into 222198-esql-control-rt

0b51062

nreese approved these changes Jun 30, 2025

View reviewed changes

Fix jest

d97cb71

Zacqary merged commit 215a4bf into elastic:main Jun 30, 2025
10 checks passed

kibanamachine added the v9.2.0 label Jun 30, 2025

kibanamachine mentioned this pull request Jun 30, 2025

[9.1] [ES|QL Controls] Refresh "Values from a query" options on dashboard reload (#225101) #225891

Merged

Zacqary added backport:version Backport to applied version labels v9.1.0 v8.19.0 and removed backport:all-open Backport to all branches that could still receive a release labels Jun 30, 2025

Zacqary mentioned this pull request Jun 30, 2025

[8.19] [ES|QL Controls] Refresh "Values from a query" options on dashboard reload (#225101) #225903

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ES|QL Controls] Refresh "Values from a query" options on dashboard reload#225101

[ES|QL Controls] Refresh "Values from a query" options on dashboard reload#225101
Zacqary merged 55 commits intoelastic:mainfrom
Zacqary:222198-esql-control-rt

Zacqary commented Jun 24, 2025 •

edited by kibanamachine

Loading

Zacqary Jun 24, 2025

elasticmachine commented Jun 24, 2025

nreese Jun 25, 2025

Zacqary Jun 25, 2025

nreese Jun 25, 2025 •

edited

Loading

Zacqary Jun 25, 2025

nreese Jun 25, 2025 •

edited

Loading

stratoula commented Jun 25, 2025

stratoula left a comment

Uh oh!

nreese Jun 27, 2025

nreese Jun 27, 2025

Zacqary commented Jun 27, 2025

nreese Jun 27, 2025 •

edited

Loading

Zacqary Jun 30, 2025

Zacqary Jun 30, 2025

nreese left a comment

nreese Jun 30, 2025

nreese Jun 30, 2025

Uh oh!

kibanamachine commented Jun 30, 2025

elasticmachine commented Jun 30, 2025

kibanamachine commented Jun 30, 2025

kibanamachine commented Jun 30, 2025

kibanamachine commented Jun 30, 2025

kibanamachine commented Jun 30, 2025

Zacqary commented Jun 30, 2025

Labels

6 participants

Conversation

Zacqary commented Jun 24, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Choose a reason for hiding this comment

elasticmachine commented Jun 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nreese Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nreese Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

stratoula commented Jun 25, 2025

stratoula left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zacqary commented Jun 27, 2025

nreese Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nreese left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Jun 30, 2025

elasticmachine commented Jun 30, 2025

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

History

kibanamachine commented Jun 30, 2025

💔 Some backports could not be created

Manual backport

Questions ?

kibanamachine commented Jun 30, 2025

kibanamachine commented Jun 30, 2025

kibanamachine commented Jun 30, 2025

Zacqary commented Jun 30, 2025

💚 All backports created successfully

Questions ?

Labels

6 participants

Zacqary commented Jun 24, 2025 •

edited by kibanamachine

Loading

nreese Jun 25, 2025 •

edited

Loading

nreese Jun 25, 2025 •

edited

Loading

nreese Jun 27, 2025 •

edited

Loading