[Data view] Fix allow_hidden usage in the request for fields#217628
Merged
kertal merged 7 commits intoelastic:mainfrom Apr 15, 2025
Merged
[Data view] Fix allow_hidden usage in the request for fields#217628kertal merged 7 commits intoelastic:mainfrom
kertal merged 7 commits intoelastic:mainfrom
Conversation
Member
Author
|
/ci |
kertal
added
Team:DataDiscovery
kertal
force-pushed
the
data-view-hidden-index-fix
branch
from
72207f1 to
bd38beb
Compare
Member
Author
|
/ci |
2 similar comments
Member
Author
|
/ci |
Member
Author
|
/ci |
kertal
force-pushed
the
data-view-hidden-index-fix
branch
from
730b3ae to
a78912a
Compare
Member
Author
|
/ci |
kertal
commented
Apr 10, 2025
|
|
||
| jest.mock('./fetcher'); | ||
|
|
||
| describe('IndexPatternsApiServer - getFieldsForWildcard', () => { |
Member
Author
There was a problem hiding this comment.
adding basic test coverage for the actual problem to fix, there could and should be much more in this area, but where to start? So I decided to focus on the actual issue
kertal
commented
Apr 10, 2025
|
|
||
| jest.mock('../../fetcher'); | ||
|
|
||
| describe('handler', () => { |
Member
Author
There was a problem hiding this comment.
there could be much more tests here, but just started with a test to catch the issue
kertal
commented
Apr 10, 2025
| >, | ||
| isRollupsEnabled: () => boolean | ||
| ) => { | ||
| export const registerFields = (router: IRouter, isRollupsEnabled: () => boolean) => { |
Member
Author
There was a problem hiding this comment.
a tiny cleanup, removing redundant code
kertal
commented
Apr 10, 2025
| allow_no_indices: allowNoIndex || false, | ||
| includeUnmapped, | ||
| }, | ||
| allowHidden, |
kertal
commented
Apr 10, 2025
| includeEmptyFields, | ||
| abortSignal, | ||
| runtimeMappings, | ||
| allowHidden, |
Contributor
|
Pinging @elastic/kibana-data-discovery (Team:DataDiscovery) |
Contributor
💚 Build Succeeded
Metrics [docs]
History
cc @kertal |
jughosta
approved these changes
Apr 15, 2025
Contributor
jughosta
left a comment
jughosta
left a comment
There was a problem hiding this comment.
LGTM 👍
Thanks for fixing it!
Contributor
|
Starting backport for target branches: 8.16, 8.17, 8.18, 8.x |
Contributor
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
This was referenced Apr 15, 2025
kertal
added a commit
to kertal/kibana
that referenced
this pull request
Apr 15, 2025
…#217628) This PR fixes the missing allowHidden/allow_hidden usage for data views using the "allow hidden and system indices" functionality when creating a data view. There are 2 endpoints being requested in Discover: `/fields` & `/fields_for_wildcards` Both use `allow_hidden=true`, but `/fields` ignored this parameter internally. This was fixed. (cherry picked from commit c25d627) # Conflicts: # src/platform/plugins/shared/data_views/server/rest_api_routes/internal/fields.ts
kertal
added a commit
to kertal/kibana
that referenced
this pull request
Apr 15, 2025
…#217628) This PR fixes the missing allowHidden/allow_hidden usage for data views using the "allow hidden and system indices" functionality when creating a data view. There are 2 endpoints being requested in Discover: `/fields` & `/fields_for_wildcards` Both use `allow_hidden=true`, but `/fields` ignored this parameter internally. This was fixed. (cherry picked from commit c25d627) # Conflicts: # src/platform/plugins/shared/data_views/server/rest_api_routes/internal/fields.ts
kertal
added a commit
to kertal/kibana
that referenced
this pull request
Apr 15, 2025
…#217628) This PR fixes the missing allowHidden/allow_hidden usage for data views using the "allow hidden and system indices" functionality when creating a data view. There are 2 endpoints being requested in Discover: `/fields` & `/fields_for_wildcards` Both use `allow_hidden=true`, but `/fields` ignored this parameter internally. This was fixed. (cherry picked from commit c25d627) # Conflicts: # src/platform/plugins/shared/data_views/server/rest_api_routes/internal/fields.ts
kertal
added a commit
that referenced
this pull request
Apr 15, 2025
…217628) (#218246) # Backport This will backport the following commits from `main` to `9.0`: - [[Data view] Fix allow_hidden usage in the request for fields (#217628)](#217628) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Matthias Wilhelm","email":"matthias.wilhelm@elastic.co"},"sourceCommit":{"committedDate":"2025-04-15T10:11:00Z","message":"[Data view] Fix allow_hidden usage in the request for fields (#217628)\n\nThis PR fixes the missing allowHidden/allow_hidden usage for data views using the \"allow hidden and system indices\" functionality when creating a data view. There are 2 endpoints being requested in Discover: `/fields` & `/fields_for_wildcards`\n\nBoth use `allow_hidden=true`, but `/fields` ignored this parameter internally. This was fixed.","sha":"c25d62739e08d114fcdec495924196a6b9a74fc4","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Feature:Data Views","Team:DataDiscovery","backport:prev-major","v9.1.0"],"title":"[Data view] Fix allow_hidden usage in the request for fields","number":217628,"url":"https://github.com/elastic/kibana/pull/217628","mergeCommit":{"message":"[Data view] Fix allow_hidden usage in the request for fields (#217628)\n\nThis PR fixes the missing allowHidden/allow_hidden usage for data views using the \"allow hidden and system indices\" functionality when creating a data view. There are 2 endpoints being requested in Discover: `/fields` & `/fields_for_wildcards`\n\nBoth use `allow_hidden=true`, but `/fields` ignored this parameter internally. This was fixed.","sha":"c25d62739e08d114fcdec495924196a6b9a74fc4"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/217628","number":217628,"mergeCommit":{"message":"[Data view] Fix allow_hidden usage in the request for fields (#217628)\n\nThis PR fixes the missing allowHidden/allow_hidden usage for data views using the \"allow hidden and system indices\" functionality when creating a data view. There are 2 endpoints being requested in Discover: `/fields` & `/fields_for_wildcards`\n\nBoth use `allow_hidden=true`, but `/fields` ignored this parameter internally. This was fixed.","sha":"c25d62739e08d114fcdec495924196a6b9a74fc4"}}]}] BACKPORT-->
kertal
added a commit
that referenced
this pull request
Apr 15, 2025
…217628) (#218250) # Backport This will backport the following commits from `main` to `8.x`: - [[Data view] Fix allow_hidden usage in the request for fields (#217628)](#217628) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Matthias Wilhelm","email":"matthias.wilhelm@elastic.co"},"sourceCommit":{"committedDate":"2025-04-15T10:11:00Z","message":"[Data view] Fix allow_hidden usage in the request for fields (#217628)\n\nThis PR fixes the missing allowHidden/allow_hidden usage for data views using the \"allow hidden and system indices\" functionality when creating a data view. There are 2 endpoints being requested in Discover: `/fields` & `/fields_for_wildcards`\n\nBoth use `allow_hidden=true`, but `/fields` ignored this parameter internally. This was fixed.","sha":"c25d62739e08d114fcdec495924196a6b9a74fc4","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Feature:Data Views","Team:DataDiscovery","backport:prev-major","v9.1.0"],"title":"[Data view] Fix allow_hidden usage in the request for fields","number":217628,"url":"https://github.com/elastic/kibana/pull/217628","mergeCommit":{"message":"[Data view] Fix allow_hidden usage in the request for fields (#217628)\n\nThis PR fixes the missing allowHidden/allow_hidden usage for data views using the \"allow hidden and system indices\" functionality when creating a data view. There are 2 endpoints being requested in Discover: `/fields` & `/fields_for_wildcards`\n\nBoth use `allow_hidden=true`, but `/fields` ignored this parameter internally. This was fixed.","sha":"c25d62739e08d114fcdec495924196a6b9a74fc4"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/217628","number":217628,"mergeCommit":{"message":"[Data view] Fix allow_hidden usage in the request for fields (#217628)\n\nThis PR fixes the missing allowHidden/allow_hidden usage for data views using the \"allow hidden and system indices\" functionality when creating a data view. There are 2 endpoints being requested in Discover: `/fields` & `/fields_for_wildcards`\n\nBoth use `allow_hidden=true`, but `/fields` ignored this parameter internally. This was fixed.","sha":"c25d62739e08d114fcdec495924196a6b9a74fc4"}}]}] BACKPORT-->
Contributor
|
This PR didn't make it into the latest BC. Updating the labels. |
kibanamachine
added
the
backport missing
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
3 similar comments
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
kertal
added a commit
that referenced
this pull request
Apr 22, 2025
…217628) (#218253) # Backport This will backport the following commits from `main` to `8.18`: - [[Data view] Fix allow_hidden usage in the request for fields (#217628)](#217628) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Matthias Wilhelm","email":"matthias.wilhelm@elastic.co"},"sourceCommit":{"committedDate":"2025-04-15T10:11:00Z","message":"[Data view] Fix allow_hidden usage in the request for fields (#217628)\n\nThis PR fixes the missing allowHidden/allow_hidden usage for data views using the \"allow hidden and system indices\" functionality when creating a data view. There are 2 endpoints being requested in Discover: `/fields` & `/fields_for_wildcards`\n\nBoth use `allow_hidden=true`, but `/fields` ignored this parameter internally. This was fixed.","sha":"c25d62739e08d114fcdec495924196a6b9a74fc4","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Feature:Data Views","Team:DataDiscovery","backport:prev-major","v9.1.0"],"title":"[Data view] Fix allow_hidden usage in the request for fields","number":217628,"url":"https://github.com/elastic/kibana/pull/217628","mergeCommit":{"message":"[Data view] Fix allow_hidden usage in the request for fields (#217628)\n\nThis PR fixes the missing allowHidden/allow_hidden usage for data views using the \"allow hidden and system indices\" functionality when creating a data view. There are 2 endpoints being requested in Discover: `/fields` & `/fields_for_wildcards`\n\nBoth use `allow_hidden=true`, but `/fields` ignored this parameter internally. This was fixed.","sha":"c25d62739e08d114fcdec495924196a6b9a74fc4"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/217628","number":217628,"mergeCommit":{"message":"[Data view] Fix allow_hidden usage in the request for fields (#217628)\n\nThis PR fixes the missing allowHidden/allow_hidden usage for data views using the \"allow hidden and system indices\" functionality when creating a data view. There are 2 endpoints being requested in Discover: `/fields` & `/fields_for_wildcards`\n\nBoth use `allow_hidden=true`, but `/fields` ignored this parameter internally. This was fixed.","sha":"c25d62739e08d114fcdec495924196a6b9a74fc4"}}]}] BACKPORT--> Co-authored-by: Julia Rechkunova <julia.rechkunova@elastic.co>
kibanamachine
added
v8.18.1
and removed
backport missing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes the missing allowHidden/allow_hidden usage for data views using the "allow hidden and system indices" functionality. This enables an advanced use case in Discover, allowing you to e.g. look at internal hidden indices like kibana event logs
When using Discover there are 2 requests for fields using 2 different endpoints, one for all fields, one for the fields that actually contain data.
/fields/fields_for_wildcardsBoth use
allow_hidden=truefor the described case, but/fieldsignored this parameter internally. This leads to issues that are resolved with this PR. However we should aim to refactor this very similar endpoints logic, 2 sources of truth here, with lack of unit test coverage (this PR is a start). This can lead to issues in this area, demonstrated by the 2 issues that are fixed.fixes #217438
fixes #212933
Testing
Create a
*kibana-event-log-ds-2025*data view with enabling "Allow hidden and system indices" in DiscoverIn Discover now should everything should work as expected, before there were just unmapped fields in the fieldlist
Checklist