[ES|QL] Interactive LOOKUP JOIN indices with editor flyout

[darnautov]

Summary

Closes #207330

This PR adds multiple enhancements to the LOOKUP JOIN UX in Discover.

⚠️ Note for reviewers: To fix multiple circular dependency issues, we had to update some plugins/packages — that’s why more codeowners are being triggered than expected.

ES|QL Editor

• Index names in LOOKUP JOIN now have a custom decorator indicating they’re interactive.
• On hover, users see a context action based on privileges:
  • Edit lookup index — shown when the index exists and the user has write privileges. Opens the flyout in edit mode.
  • View lookup index — shown when the index exists and the user has read-only (with view_index_metadata) privileges. Opens the flyout in read-only mode.
  • Create lookup index — shown when the index doesn’t exist but the user has create-index privileges for the pattern. Opens the flyout in create mode with an empty state.
• Autocomplete now surfaces a “Create lookup index <index_input>” suggestion at the top when the user has the required privileges.

---

Index Editor flyout

The index editor lives in a dedicated package, kbn-index-editor, which provides:

• A flyout for create/view/edit.
• The UI action to open the flyout (registered by the esql plugin).

Key features

Inline editing

Built with the UnifiedDataTable component with some changes:

• Click any cell to open an edit popover with basic client-side validation.
• Add column lets users define a new field by name; Elasticsearch infers the data type on save.
• For empty indices or those with fewer than 4 fields, placeholder columns are appended to make defining new fields easier.
• Add row inserts an empty row at the top.
• Save changes batches edits and persists them via the bulk_update API. On success, the flyout closes; on error, a message appears at the top of the flyout.

Update/create from file

The flyout integrates with the file upload plugin to:

• Create an index from CSV files.
• Append documents to an existing index.
• Support multiple files per upload (click Upload file or drag-and-drop anywhere in the flyout).
• Preview file contents before uploading.
• Show overall upload progress; once complete, the regular grid renders documents from the index.

Query bar

• The query bar accepts the ES|QL query string format: https://www.elastic.co/docs/reference/query-languages/esql/functions-operators/search-functions#esql-qstr.
• Query this index opens Discover in a new tab with a prepopulated ES|QL query.

---

Modes & behavior

Depending on how the flyout is opened:

Create mode

• Set the index name inline in the flyout header. If the user typed an index name in the ES|QL editor, it’s prefilled (and editable).
• Start from an empty state: define columns and add rows manually or upload files.
• The entire flyout acts as a file drop zone.

Edit mode

• Edit any document in the table.
• Add new columns and rows.
• Upload files to append documents.

View mode

• Disabled actions:
  • File upload and drag-and-drop
  • Inline editing in the data grid

---

Permissions & visibility

Actions are gated by:

• Write privileges (edit).
• Read-only with view_index_metadata (view).
• Create-index privileges on the pattern (create).

---

How to test

1. Open Discover with ES|QL mode.
2. Type a LOOKUP JOIN referencing:
   • an existing index you can write to,
   • an existing index you can only read,
   • and a non-existent index matching a pattern you can create.
3. Hover each index name and verify the correct action label.
4. Open the flyout:
   • Edit: modify a few cells, add a column and a row, click Save changes, confirm success closes the flyout and data persists.
   • Create: set an index name, add columns/rows, or upload a CSV; save and verify the index is created.
   • View: confirm editing and uploads are disabled.
5. Use Upload file (and drag-and-drop) with multiple CSVs; verify preview, progress, and final grid.
6. Enter a query in the query bar and click Query this index; confirm Discover opens with the prefilled ES|QL.

---

Release note

Improve Discover’s LOOKUP JOIN UX with an index editor flyout supporting inline editing, CSV import, contextual actions based on privileges, and a query bar that opens Discover with a prefilled ES|QL query.

Checklist

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

[jughosta]

The search requires Enter keystroke. Might become confusing for some users that it does not filter right away and there is no button to submit the search.

[jughosta]

Also, could be helpful to make the search case insensitive.

[sddonne]

The file uploader creates the string field as keywords, so it's not allowing to perform case insensitive searches, will check if it's possible to change it.

With regard the onEnter behaviour, whats your take about it @ninoslavmiskovic ?

[ninoslavmiskovic]

I also mentioned the case insensitive, but that is something that will require some additional work, so perhaps a separate PR.

Regarding the search, I don't see it as a blocker and it follows the EUI component. We can monitor feedback post launch, but I am not that concern since e.g. people are used to google.com and there is no search button here only "enter" for search.

[jughosta]

Found a couple of issues related to sorting:

1. when in edit mode I add a new row, sorting does not account for their values
2. when I after that add more empty rows and then try to delete them, my existing rows gets deleted instead

[sddonne]

Sorting is being done server side, thats why unsaved values do not participate in it.

With regard the deletion bug, thanks for spotting it, it's happening because both, our sorting and the UnifiedDataTable inner sorting are being executed at the same time, leading to an inconsistent state.

It seems there is no way to disable the default sorting mechanism in UnifiedDataTable, talking with @darnautov we decided to disable the sorting functionality for now, as it's not critical for the feature and it's not trivial to solve.

CC: @jughosta @stratoula

[stratoula]

I agree with this decision, let's create an issue to track it

[sddonne]

198142c

[nreese]

kibana-presentation changes LGTM - small changes in file upload
code review only

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 2a133c6
• Cloud Deployment

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
console	246	247	+1
controls	397	398	+1
dashboard	720	721	+1
data	516	517	+1
datasetQuality	844	845	+1
dataVisualizer	763	764	+1
discover	1553	1554	+1
esql	131	418	+287
eventAnnotationListing	567	568	+1
infra	1568	1569	+1
lens	1412	1417	+5
lists	336	337	+1
maps	1284	1285	+1
ml	2516	2517	+1
observability	1405	1406	+1
observabilityAIAssistantApp	441	442	+1
onechat	506	507	+1
securitySolution	7993	7994	+1
slo	1249	1250	+1
stackAlerts	282	304	+22
streamsApp	648	649	+1
triggersActionsUi	972	973	+1
unifiedSearch	462	463	+1
total			+334

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/discover-utils	355	363	+8
@kbn/es-types	31	32	+1
@kbn/esql-utils	102	103	+1
@kbn/esql-validation-autocomplete	29	33	+4
@kbn/file-upload	81	82	+1
@kbn/index-editor	-	19	+19
@kbn/monaco	193	197	+4
@kbn/unified-data-table	112	113	+1
total			+39

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
cloudSecurityPosture	565.0KB	565.8KB	+806.0B
dashboard	646.1KB	646.1KB	+17.0B
dataVisualizer	684.1KB	684.6KB	+457.0B
discover	1.1MB	1.1MB	+839.0B
esql	270.9KB	519.0KB	⚠️ +248.1KB
esqlDataGrid	145.0KB	145.8KB	+777.0B
lens	1.5MB	1.5MB	+241.0B
maps	3.1MB	3.1MB	+17.0B
observabilityAIAssistantApp	177.7KB	177.7KB	+17.0B
onechat	257.6KB	257.6KB	+41.0B
securitySolution	10.5MB	10.5MB	+805.0B
slo	974.2KB	975.0KB	+807.0B
stackAlerts	64.9KB	64.9KB	+20.0B
total			+252.8KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
dataVisualizer	26.4KB	26.5KB	+76.0B
esql	9.3KB	11.0KB	+1.7KB
esqlDataGrid	9.1KB	9.1KB	+28.0B
fileUpload	15.8KB	16.0KB	+214.0B
kbnUiSharedDeps-srcJs	3.8MB	3.8MB	+1.1KB
lens	60.9KB	60.9KB	+17.0B
total			+3.1KB

Unknown metric groups

API count

id	before	after	diff
@kbn/cell-actions	44	45	+1
@kbn/discover-utils	413	421	+8
@kbn/es-types	31	32	+1
@kbn/esql-utils	120	123	+3
@kbn/esql-validation-autocomplete	32	36	+4
@kbn/file-upload	81	82	+1
@kbn/index-editor	-	20	+20
@kbn/monaco	193	197	+4
@kbn/unified-data-table	121	136	+15
total			+57

async chunk count

id	before	after	diff
esql	7	12	+5

ESLint disabled line counts

id	before	after	diff
@kbn/esql-editor	6	7	+1
@kbn/index-editor	-	1	+1
total			+2

Total ESLint disabled count

id	before	after	diff
@kbn/esql-editor	6	7	+1
@kbn/index-editor	-	1	+1
total			+2

History

• 💚 Build #339174 succeeded b676f28
• 💔 Build #338933 failed 24ecbb3
• 💔 Build #338033 failed 673425b
• 💚 Build #337957 succeeded ef47ffd
• 💔 Build #337773 failed 868a89d

cc @darnautov @sddonne

[jughosta]

Looks good from my side, thanks for addressing the comments!

[jughosta]

Thanks!
Would be great if it were preselecting this tab or having a red badge next to the tab name. Otherwise, it's hard to notice the error message.

[jughosta]

thanks!

[akowalska622]

I've done an additional round of local testing and didn't spot anything wrong ✅