[Reporting] roll over the reporting data stream when the template has changed by pmuellr · Pull Request #234119 · elastic/kibana

pmuellr · 2025-09-05T04:32:36Z

This PR adds code run at startup to check the version in the reporting index template against the _meta.template_version value stamped into each backing index's mappings via elastic/elasticsearch#133846

If it determines there are existing reporting indices that have not been created with the current index template version, it will roll over the reporting data stream, to ensure future indexing will use the latest mappings. This is done with the "lazy" option, which will perform the roll over on the next write to the datastream. This will prevent situations where multiple Kibanas restarting at the same time would roll over multiple times.

Note that we added a function test here, but would have preferred a jest integration test, but we can't. Instead, we expose the stand-alone-ish rollDataStreamIfRequired() function via an HTTP route added in the function tests. Then invoke it via that route in the function tests. This allows us to set up several scenarios with real data streams, templates, and mappings, to ensure the correct behavior.

to validate

Start with an old Kibana, like 8.19, which we will "upgrade" to this PR running ES with -E path.data= to re-use the "old" resources when upgrading.

Generate a report. I always load sample data, go it's dashboard, and generate a PDF.

Check the index template, and you should see a version less than 18. http://localhost:5601/app/management/data/index_management/templates/.kibana-reporting

Kill that ES and Kibana, and start them on this PR.

The logs should show the message: [INFO ][plugins.reporting.store] Data stream .kibana-reporting has no mapping versions so rolling over the data stream

Checking the index template again, and you should see the version is now 18. You should also see in the mappings "_meta": { "template_version": 18 }

Release note

Startup processing for reporting has changed to roll over the reporting data stream, if it's template version is newer than the version in the data stream's mappings. This fixes a problem when a new field added to the index template mappings in a new stack version, but the reporting data stream already exists, and the new mappings are not automatically applied. By rolling over the data stream, if required, we will ensure the latest backing index of the data stream will match the template's mappings.

Copilot

Pull Request Overview

This PR implements automatic data stream rollover for Kibana reporting indices when the index template has changed. The solution addresses a bug where new fields added to index templates (like space_id) weren't being applied to existing indices, causing reports in non-default spaces to not appear properly in the reports list.

Key changes:

Added startup logic to compare index template versions with backing index mapping versions
Automatically rolls over the reporting data stream when template updates are detected
Enhanced error handling and logging for the startup process

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
x-pack/platform/plugins/private/reporting/server/lib/store/store.ts	Implements rollover logic with version comparison and integrates it into the startup process
src/platform/packages/private/kbn-reporting/server/constants.ts	Adds constants for index template name and mapping meta field

x-pack/platform/plugins/private/reporting/server/lib/store/store.ts

+      // newer mappings than the template shouldn't happen
+      throw new Error(`${msgPrefix} has newer mappings than the template`);
+    } else {
+      // lastest mappings already applied


… changed resolves: elastic#231200 This PR adds code run at startup to check the version in the reporting index template against the `_meta.template_version` value stamped into each backing index's mappings via elastic/elasticsearch#133846 If it determines there are existing reporting indices that have not been created with the current index template version, it will roll over the reporting data stream, to ensure future indexing will use the latest mappings. Startup processing for reporting has changed to apply the current index template mappings to the kibana reporting index, if it exists. This fixes a problem where a new field added to the index template, however it was not applied, so the field could not reliably used. The field was `space_id`, which could cause reports in non-default spaces to not appear in the reports list.

elasticmachine · 2025-09-19T16:23:36Z

Pinging @elastic/response-ops (Team:ResponseOps)

Copilot

Pull Request Overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 2 comments.

pmuellr · 2025-09-19T18:35:46Z

x-pack/platform/test/reporting_api_integration/reporting_without_security/roll_datastream.ts

+  // @ts-ignore ; I've seen either result.acknowledged or result.status === 404
+  expect(result.acknowledged || result.status === 404).to.be(true);


It's a test, not adding that kinda stuff for a test! Not quite sure why the typing was off on that API, but I did see a status: 404 in the debugger at one point, so made sure we checked that ...

Do you think Copilot will request changes to block you from merging? 😂

hehe - actually, I'm pretty sure it explicitly says such comments will never block a merge.

pmuellr · 2025-09-19T18:38:51Z

x-pack/platform/plugins/private/reporting/server/lib/store/rollover.ts

+  // get the value of _meta.template_version from each index's mappings
+  const mappingsVersions = mappingsArray
+    .map((m) => m.mappings._meta?.[REPORTING_INDEX_TEMPLATE_MAPPING_META_FIELD])
+    .filter((a: any): a is number => typeof a === 'number');


I think a is fine here. I probably should have mentioned WHY I'm doing this. In case one of the numbers isn't a number (like undefined), we want to filter them out, but you'd think you could do

.filter((a: any) => typeof a === 'number')

The problem is the result of that is typed any[]. Adding the predicate : a is number ends up typing the result as number[].

ymao1

LGTM! Works as described.

Verified by doing the following:

started on 8.18, stack template version 14, created a report
upgraded to latest 8.19, stack template version 18, created a report and see the bad space_id field dynamically mapped
upgraded to this PR, stack template version 18, see startup messages rolling over the reporting index, created a report, see the rolled over index with correct mappings
restarted Kibana, stack template version 18, see no rollover
added new stack template version 19 locally and run ES from source, see startup messages rolling over the reporting index, created a report, see the rolled over index with new mapping.

pmuellr · 2025-09-23T02:57:11Z

/ci

pmuellr · 2025-09-23T02:59:28Z

@elasticmachine merge upstream

csr

x-pack/platform/test/tsconfig.json changes LGTM 👍

pmuellr · 2025-09-25T13:32:45Z

@elasticmachine merge upstream

pmuellr · 2025-09-25T19:42:14Z

@elasticmachine merge upstream

kibanamachine · 2025-09-29T23:06:59Z

Starting backport for target branches: 8.18, 8.19, 9.0, 9.1

https://github.com/elastic/kibana/actions/runs/18113228075

elasticmachine · 2025-09-29T23:07:49Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: fc0482c

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #106 / Endpoint plugin @ess @serverless @skipInServerlessMKI When attempting to call an endpoint api "after all" hook in "@ess @serverless @skipInServerlessMKI When attempting to call an endpoint api"
[job] [logs] FTR Configs #88 / Space awareness actions POST /agents/actions/{actionId}/cancel should return 404 if the action is in a different space

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/reporting-server`	91	93	+2

Unknown metric groups

API count

id	before	after	diff
`@kbn/reporting-server`	91	93	+2

History

💔 Build #344690 failed 40a57f3
💔 Build #344007 failed 612b2a4
💔 Build #343817 failed 39725b7
💛 Build #342365 was flaky 6ec4ada
💔 Build #341559 failed e2cedac

kibanamachine · 2025-09-29T23:16:11Z

💔 All backports failed

Status	Branch	Result
❌	8.18	Backport failed because of merge conflicts
❌	8.19	Backport failed because of merge conflicts
❌	9.0	Backport failed because of merge conflicts
❌	9.1	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 234119

Questions ?

Please refer to the Backport tool documentation

… changed (#234119) resolves: #231200 This PR adds code run at startup to check the version in the reporting index template against the `_meta.template_version` value stamped into each backing index's mappings via elastic/elasticsearch#133846 If it determines there are existing reporting indices that have not been created with the current index template version, it will roll over the reporting data stream, to ensure future indexing will use the latest mappings. This is done with the "lazy" option, which will perform the roll over on the next write to the datastream. This will prevent situations where multiple Kibanas restarting at the same time would roll over multiple times. --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

… changed (elastic#234119) resolves: elastic#231200 This PR adds code run at startup to check the version in the reporting index template against the `_meta.template_version` value stamped into each backing index's mappings via elastic/elasticsearch#133846 If it determines there are existing reporting indices that have not been created with the current index template version, it will roll over the reporting data stream, to ensure future indexing will use the latest mappings. This is done with the "lazy" option, which will perform the roll over on the next write to the datastream. This will prevent situations where multiple Kibanas restarting at the same time would roll over multiple times. --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

kibanamachine · 2025-09-30T23:48:38Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 234119 locally
cc: @pmuellr

… changed (elastic#234119) resolves: elastic#231200 This PR adds code run at startup to check the version in the reporting index template against the `_meta.template_version` value stamped into each backing index's mappings via elastic/elasticsearch#133846 If it determines there are existing reporting indices that have not been created with the current index template version, it will roll over the reporting data stream, to ensure future indexing will use the latest mappings. This is done with the "lazy" option, which will perform the roll over on the next write to the datastream. This will prevent situations where multiple Kibanas restarting at the same time would roll over multiple times. --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit c89ba82) # Conflicts: # .github/CODEOWNERS # x-pack/platform/test/reporting_api_integration/reporting_without_security.config.ts # x-pack/platform/test/tsconfig.json

… changed (elastic#234119) resolves: elastic#231200 This PR adds code run at startup to check the version in the reporting index template against the `_meta.template_version` value stamped into each backing index's mappings via elastic/elasticsearch#133846 If it determines there are existing reporting indices that have not been created with the current index template version, it will roll over the reporting data stream, to ensure future indexing will use the latest mappings. This is done with the "lazy" option, which will perform the roll over on the next write to the datastream. This will prevent situations where multiple Kibanas restarting at the same time would roll over multiple times. --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit c89ba82) # Conflicts: # .github/CODEOWNERS # x-pack/test/reporting_api_integration/reporting_without_security.config.ts # x-pack/test/reporting_api_integration/reporting_without_security/index.ts # x-pack/test/reporting_api_integration/reporting_without_security/roll_datastream.ts # x-pack/test/tsconfig.json

… changed (elastic#234119) resolves: elastic#231200 This PR adds code run at startup to check the version in the reporting index template against the `_meta.template_version` value stamped into each backing index's mappings via elastic/elasticsearch#133846 If it determines there are existing reporting indices that have not been created with the current index template version, it will roll over the reporting data stream, to ensure future indexing will use the latest mappings. This is done with the "lazy" option, which will perform the roll over on the next write to the datastream. This will prevent situations where multiple Kibanas restarting at the same time would roll over multiple times. --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit c89ba82) # Conflicts: # .github/CODEOWNERS # x-pack/platform/test/reporting_api_integration/reporting_without_security.config.ts # x-pack/platform/test/tsconfig.json

pmuellr · 2025-10-01T12:49:34Z

💚 All backports created successfully

Status	Branch	Result
✅	9.1
✅	9.0
✅	8.19
✅	8.18

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

… changed (elastic#234119) resolves: elastic#231200 This PR adds code run at startup to check the version in the reporting index template against the `_meta.template_version` value stamped into each backing index's mappings via elastic/elasticsearch#133846 If it determines there are existing reporting indices that have not been created with the current index template version, it will roll over the reporting data stream, to ensure future indexing will use the latest mappings. This is done with the "lazy" option, which will perform the roll over on the next write to the datastream. This will prevent situations where multiple Kibanas restarting at the same time would roll over multiple times. --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit c89ba82) # Conflicts: # .github/CODEOWNERS # x-pack/test/reporting_api_integration/reporting_without_security.config.ts # x-pack/test/reporting_api_integration/reporting_without_security/index.ts # x-pack/test/reporting_api_integration/reporting_without_security/roll_datastream.ts # x-pack/test/tsconfig.json

pmuellr · 2025-10-01T19:58:26Z

We only need backports to 8.19 and 9.1, based on the pre-req PR elastic/elasticsearch#133846 only being backported there.

kibanamachine · 2025-10-01T20:01:22Z

Starting backport for target branches: 8.19, 9.1

https://github.com/elastic/kibana/actions/runs/18174039913

kibanamachine · 2025-10-01T20:08:39Z

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts
❌	9.1	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 234119

Questions ?

Please refer to the Backport tool documentation

…ate has changed (#234119) (#237115) # Backport This will backport the following commits from `main` to `8.19`: - [[Reporting] roll over the reporting data stream when the template has changed (#234119)](#234119)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…te has changed (#234119) (#237049) # Backport This will backport the following commits from `main` to `9.1`: - [[Reporting] roll over the reporting data stream when the template has changed (#234119)](#234119)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

pmuellr · 2025-10-02T13:43:19Z

This PR is a bit noisy, with failed auto-merged back ports, so I thought I'd be specific about the ones that actually merged:

[8.19] [Reporting] roll over the reporting data stream when the template has changed (#234119) #237115
[9.1] [Reporting] roll over the reporting data stream when the template has changed (#234119) #237049

… changed (elastic#234119) resolves: elastic#231200 This PR adds code run at startup to check the version in the reporting index template against the `_meta.template_version` value stamped into each backing index's mappings via elastic/elasticsearch#133846 If it determines there are existing reporting indices that have not been created with the current index template version, it will roll over the reporting data stream, to ensure future indexing will use the latest mappings. This is done with the "lazy" option, which will perform the roll over on the next write to the datastream. This will prevent situations where multiple Kibanas restarting at the same time would roll over multiple times. --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

pmuellr requested a review from Copilot September 5, 2025 14:56

Copilot AI reviewed Sep 5, 2025

View reviewed changes

pmuellr force-pushed the 231200-reporting-roll-over-ds-2 branch from c1a75f7 to 309e269 Compare September 8, 2025 23:54

pmuellr force-pushed the 231200-reporting-roll-over-ds-2 branch from 9b1a6c7 to ab475ee Compare September 18, 2025 14:13

pmuellr added 2 commits September 18, 2025 18:31

fix jest test

37cacbd

pmuellr force-pushed the 231200-reporting-roll-over-ds-2 branch from ab6424e to 37cacbd Compare September 18, 2025 22:33

pmuellr added Team:ResponseOps Platform ResponseOps team (formerly the Cases and Alerting teams) t// Feature:Reporting:Framework Reporting issues pertaining to the overall framework release_note:fix backport:all-open Backport to all branches that could still receive a release labels Sep 19, 2025

remove (hopefully) unneccessary retry in function test

e2cedac

pmuellr marked this pull request as ready for review September 19, 2025 16:23

pmuellr requested review from a team as code owners September 19, 2025 16:23

pmuellr marked this pull request as draft September 19, 2025 17:16

pmuellr marked this pull request as ready for review September 19, 2025 17:17

pmuellr requested a review from Copilot September 19, 2025 17:18

Copilot AI reviewed Sep 19, 2025

View reviewed changes

ymao1 approved these changes Sep 22, 2025

View reviewed changes

Merge branch 'main' into 231200-reporting-roll-over-ds-2

6ec4ada

csr approved these changes Sep 24, 2025

View reviewed changes

Merge branch 'main' into 231200-reporting-roll-over-ds-2

f07525c

pmuellr enabled auto-merge (squash) September 25, 2025 13:50

[CI] Auto-commit changed files from 'node scripts/generate codeowners'

39725b7

pmuellr merged commit c89ba82 into elastic:main Sep 29, 2025
14 checks passed

kibanamachine added the v9.2.0 label Sep 29, 2025

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Sep 30, 2025

pmuellr mentioned this pull request Oct 1, 2025

[9.1] [Reporting] roll over the reporting data stream when the template has changed (#234119) #237049

Merged

pmuellr mentioned this pull request Oct 1, 2025

[9.0] [Reporting] roll over the reporting data stream when the template has changed (#234119) #237112

Closed

pmuellr mentioned this pull request Oct 1, 2025

[8.19] [Reporting] roll over the reporting data stream when the template has changed (#234119) #237115

Merged

pmuellr mentioned this pull request Oct 1, 2025

[8.18] [Reporting] roll over the reporting data stream when the template has changed (#234119) #237118

Closed

pmuellr added backport:version Backport to applied version labels v8.19.5 v9.1.5 and removed backport:all-open Backport to all branches that could still receive a release labels Oct 1, 2025

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Oct 2, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Reporting] roll over the reporting data stream when the template has changed#234119

[Reporting] roll over the reporting data stream when the template has changed#234119
pmuellr merged 9 commits intoelastic:mainfrom
pmuellr:231200-reporting-roll-over-ds-2

pmuellr commented Sep 5, 2025 •

edited by kibanamachine

Loading

Copilot AI left a comment

This comment was marked as outdated.

Uh oh!

elasticmachine commented Sep 19, 2025

Copilot AI left a comment

This comment was marked as off-topic.

Uh oh!

pmuellr Sep 19, 2025

ymao1 Sep 19, 2025

pmuellr Sep 19, 2025

This comment was marked as spam.

Uh oh!

pmuellr Sep 19, 2025

ymao1 left a comment

pmuellr commented Sep 23, 2025

pmuellr commented Sep 23, 2025

csr left a comment

pmuellr commented Sep 25, 2025

pmuellr commented Sep 25, 2025

Uh oh!

kibanamachine commented Sep 29, 2025

elasticmachine commented Sep 29, 2025

API count

kibanamachine commented Sep 29, 2025

kibanamachine commented Sep 30, 2025

pmuellr commented Oct 1, 2025

pmuellr commented Oct 1, 2025

kibanamachine commented Oct 1, 2025

kibanamachine commented Oct 1, 2025

pmuellr commented Oct 2, 2025

Labels

6 participants

		// @ts-ignore ; I've seen either result.acknowledged or result.status === 404
		expect(result.acknowledged \|\| result.status === 404).to.be(true);

Conversation

pmuellr commented Sep 5, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

to validate

Release note

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

This comment was marked as outdated.

Uh oh!

elasticmachine commented Sep 19, 2025

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

This comment was marked as off-topic.

Uh oh!

pmuellr Sep 19, 2025

Choose a reason for hiding this comment

ymao1 Sep 19, 2025

Choose a reason for hiding this comment

pmuellr Sep 19, 2025

Choose a reason for hiding this comment

This comment was marked as spam.

Uh oh!

pmuellr Sep 19, 2025

Choose a reason for hiding this comment

ymao1 left a comment

Choose a reason for hiding this comment

pmuellr commented Sep 23, 2025

pmuellr commented Sep 23, 2025

csr left a comment

Choose a reason for hiding this comment

pmuellr commented Sep 25, 2025

pmuellr commented Sep 25, 2025

Uh oh!

kibanamachine commented Sep 29, 2025

elasticmachine commented Sep 29, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

API count

History

kibanamachine commented Sep 29, 2025

💔 All backports failed

Manual backport

Questions ?

kibanamachine commented Sep 30, 2025

pmuellr commented Oct 1, 2025

💚 All backports created successfully

Questions ?

pmuellr commented Oct 1, 2025

kibanamachine commented Oct 1, 2025

kibanamachine commented Oct 1, 2025

💔 All backports failed

Manual backport

Questions ?

pmuellr commented Oct 2, 2025

Labels

6 participants

pmuellr commented Sep 5, 2025 •

edited by kibanamachine

Loading