[Fleet] Use type@lifecycle ILMs for new package installs by juliaElastic · Pull Request #241992 · elastic/kibana

juliaElastic · 2025-11-05T14:18:17Z

Summary

Closes https://github.com/elastic/ingest-dev/issues/6082

Start using type@lifecycle ILM policies when installing new packages and record migration status.

To test:

install a package e.g. system
verify that the migration status is saved
verify that the type@lifecycle ILM policies are used

[2025-11-11T10:21:53.722+01:00][INFO ][plugins.fleet] Saving ILM migration status changes: [["logs","success"],["metrics","success"],["synthetics",null]]

GET .kibana_ingest/_doc/ingest_manager_settings:fleet-default-settings

"_source": {
    "ingest_manager_settings": {
      "prerelease_integrations_enabled": true,
      "use_space_awareness_migration_status": "success",
      "ilm_migration_status": {
        "logs": "success",
        "metrics": "success"
      }
    },

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

See some risk examples
...

…atus --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/saved_objects/_import --include-path /api/saved_objects/_export --include-path /api/maintenance_window --include-path /api/agent_builder --update

…tion_tests/ci_checks

…atus --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/saved_objects/_import --include-path /api/saved_objects/_export --include-path /api/maintenance_window --include-path /api/agent_builder --update

…tion_tests/ci_checks

juliaElastic · 2025-11-12T16:05:25Z

code LGTM, @juliaElastic I am curious if we could maybe move some complexity from the template install codepath that is already complex and have a dedicated step in package install maybe something, package_install_precheck where we could build the ilm_migration status for all types, and eventually create the global component ES assets in the future, wdyt?

Good idea, we can do that. The difference would be that for each package installation we would check the migration status for all types (logs, metrics, synthetics), not only for those data stream types that are part of the package. I think it should be fine, we want to do the migration sooner than later.

azasypkin

The PR adds a new migration to the GLOBAL_SETTINGS_SAVED_OBJECT_TYPE (ingest_manager_settings) saved object type that doesn't seem to be an encrypted saved object, so LGTM from the security perspective.

nchaulet

code LGTM 🚀

…atus --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/saved_objects/_import --include-path /api/saved_objects/_export --include-path /api/maintenance_window --include-path /api/agent_builder --update

seanrathier

LGTM

x-pack/platform/plugins/shared/fleet/server/saved_objects/index.ts

…tion_tests/ci_checks

jesuswr · 2025-11-14T09:32:49Z

x-pack/platform/plugins/shared/fleet/server/types/rest_spec/settings.ts

 });

+export const SettingsSchemaV5 = schema.object({
+  has_seen_add_data_notice: schema.maybe(schema.boolean()),


There's a property defined in the mappings but not in the schema, not sure if it's intentional: fleet_server_hosts: { type: 'keyword' }

I think it's a deprecated field that's no longer used in the SettingsSchema.
It was removed from the API here: https://github.com/elastic/kibana/pull/198799/files#diff-928c69c436a8c2e0ba247288a26603a901701e6ca70b2db67a22e3ed757efb68

tiansivive

LGTM from EA

elasticmachine · 2025-11-17T10:54:51Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: d00d92a

Failed CI Steps

Scout Test Run Builder

Test Failures

[job] [logs] Scout Test Run Builder / serverless-security - EUI testing wrapper: EuiToast - toast
[job] [logs] Scout Test Run Builder / toast

Metrics [docs]

✅ unchanged

History

) ## Summary Closes elastic/ingest-dev#6082 Start using `type@lifecycle` ILM policies when installing new packages and record migration status. To test: - install a package e.g. system - verify that the migration status is saved - verify that the `type@lifecycle` ILM policies are used ``` [2025-11-11T10:21:53.722+01:00][INFO ][plugins.fleet] Saving ILM migration status changes: [["logs","success"],["metrics","success"],["synthetics",null]] GET .kibana_ingest/_doc/ingest_manager_settings:fleet-default-settings "_source": { "ingest_manager_settings": { "prerelease_integrations_enabled": true, "use_space_awareness_migration_status": "success", "ilm_migration_status": { "logs": "success", "metrics": "success" } }, ``` <img width="1658" height="813" alt="image" src="https://github.com/user-attachments/assets/750e0127-6e75-481b-a159-0e3432893ab7" /> ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>

ilm migration

ff9c221

juliaElastic added release_note:enhancement backport:version Backport to applied version labels v9.3.0 labels Nov 5, 2025

kibanamachine and others added 24 commits November 5, 2025 14:31

Changes from node scripts/check_mappings_update --fix

012fb9a

Changes from make api-docs

2deb26a

Changes from node scripts/jest_integration -u src/core/server/integra…

353bdef

…tion_tests/ci_checks

Merge branch 'main' into ilm-migration

d262580

Changes from node scripts/jest_integration -u src/core/server/integra…

e901d2e

…tion_tests/ci_checks

fix error with missing forwardCompatibility

fddfc14

Changes from node scripts/jest_integration -u src/core/server/integra…

f9a0931

…tion_tests/ci_checks

add default_settings unit tests

7e52563

add tests on save migration changes

815fb14

add more unit tests

d01bc67

getSettingsOrUndefined

6fd336e

fix checks

ace5e01

Merge branch 'main' into ilm-migration

7d03370

fix SO error

46f461b

Changes from node scripts/jest_integration -u src/core/server/integra…

dd4d7f5

…tion_tests/ci_checks

fix SO checks

eeaf5a6

skip ILM logic if disabled in serverless

368caf0

Changes from make api-docs

6ca5b00

Changes from node scripts/jest_integration -u src/core/server/integra…

57f0fd2

…tion_tests/ci_checks

add condition

119abbb

fix log

d8db68f

Merge branch 'main' into ilm-migration

cd341d1

elastic deleted a comment from kibanamachine Nov 10, 2025

fix migrate test

aa89ae2

move ilm migration check to new step in package install state machine

fc5f667

azasypkin approved these changes Nov 12, 2025

View reviewed changes

nchaulet approved these changes Nov 12, 2025

View reviewed changes

juliaElastic and others added 2 commits November 13, 2025 08:20

Merge branch 'main' into ilm-migration

e2cc18d

Changes from node scripts/check_saved_objects

7467983

elastic-vault-github-plugin-prod bot requested a review from a team as a code owner November 13, 2025 07:59

juliaElastic and others added 3 commits November 13, 2025 10:24

fix tests

0dc018e

Changes from make api-docs

5884df1

delanni approved these changes Nov 13, 2025

View reviewed changes

seanrathier reviewed Nov 13, 2025

View reviewed changes

seanrathier approved these changes Nov 13, 2025

View reviewed changes

jesuswr requested changes Nov 13, 2025

View reviewed changes

x-pack/platform/plugins/shared/fleet/server/saved_objects/index.ts Show resolved Hide resolved

juliaElastic and others added 3 commits November 14, 2025 08:28

use dynamic:false

1b59224

fix SO mappings

4484d98

Changes from node scripts/jest_integration -u src/core/server/integra…

9268f46

…tion_tests/ci_checks

jesuswr reviewed Nov 14, 2025

View reviewed changes

tiansivive approved these changes Nov 14, 2025

View reviewed changes

Merge branch 'main' into ilm-migration

d00d92a

juliaElastic requested a review from jesuswr November 17, 2025 10:12

jesuswr approved these changes Nov 17, 2025

View reviewed changes

juliaElastic merged commit 0fc9412 into elastic:main Nov 17, 2025
12 checks passed

kibanamachine added backport:skip This PR does not require backporting and removed backport:version Backport to applied version labels labels Nov 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Use type@lifecycle ILMs for new package installs#241992

[Fleet] Use type@lifecycle ILMs for new package installs#241992
juliaElastic merged 60 commits intoelastic:mainfrom
juliaElastic:ilm-migration

juliaElastic commented Nov 5, 2025 •

edited

Loading

juliaElastic commented Nov 12, 2025

azasypkin left a comment

nchaulet left a comment

seanrathier left a comment

Uh oh!

jesuswr Nov 14, 2025

juliaElastic Nov 17, 2025 •

edited

Loading

tiansivive left a comment

elasticmachine commented Nov 17, 2025

Uh oh!

Labels

11 participants

Conversation

juliaElastic commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Identify risks

juliaElastic commented Nov 12, 2025

azasypkin left a comment

Choose a reason for hiding this comment

nchaulet left a comment

Choose a reason for hiding this comment

seanrathier left a comment

Choose a reason for hiding this comment

Uh oh!

jesuswr Nov 14, 2025

Choose a reason for hiding this comment

juliaElastic Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

tiansivive left a comment

Choose a reason for hiding this comment

elasticmachine commented Nov 17, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

Uh oh!

Labels

11 participants

juliaElastic commented Nov 5, 2025 •

edited

Loading

juliaElastic Nov 17, 2025 •

edited

Loading