Skip to content

Support var removals on integration upgrade#238542

Merged
MichelLosier merged 7 commits intoelastic:mainfrom
MichelLosier:support-var-migrations-on-integration-upgrade
Oct 15, 2025
Merged

Support var removals on integration upgrade#238542
MichelLosier merged 7 commits intoelastic:mainfrom
MichelLosier:support-var-migrations-on-integration-upgrade

Conversation

@MichelLosier
Copy link
Contributor

@MichelLosier MichelLosier commented Oct 10, 2025 •
edited by kibanamachine
Loading

Summary

Resolves: #235242

  • On package policy upgrade:
    • Adds support for checking global var additions on package upgrade
    • Removes stale variables (global, input, or stream) from the original package policy that are no longer defined in the new package version.
      • This helps avoid situations where variables originally defined at a lower scope, then raised to a higher scope in a new version, don't take precedence with old values over the new definition.

Other notes:

  • This does not try to infer if a variable previously defined at one scope (global, input, or stream), has moved to a different one and migrate existing values to the new scope. I think 'best guesses' at this can lead to unexpected bugs, and if this is something we want to support we may want to look at how we can define migrations of this sort in the package spec.

To reproduce

  • Install the package "Trend Micro Vision One" at version 2.1.0
  • In the integrations repo, pull this branch from this PR https://github.com/elastic/integrations/pull/13588/files
    • This is v2.2.0 and moves some input variables to the package global scope
    • Build the package elastic-package build
  • Run package-registry locally, and mount the built packages to the container
docker run --rm -it -p 8080:8080 \
  -v /local/dir/to/integrations/build/packages:/packages/package-registry \
  $(docker images -q docker.elastic.co/package-registry/package-registry:main)
  • Update kibana.dev.yml to use local registry xpack.fleet.registryUrl: http://0.0.0.0:8080
  • Update the integration to 2.2.0 and select the option to upgrade integration policies.
  • It should fail to update the policies and require filling in the new empty fields for the policy separately

Release notes

  • Fixes on integration policy upgrade:
    • New package global vars not being included
    • Stale variable references not being removed

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

  • Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
  • Documentation was added for features that require explanation or tutorials
  • Unit or functional tests were updated or added to match the most common scenarios
  • If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
  • This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
  • Flaky Test Runner was used on any tests changed
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
  • Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.
@MichelLosier
Copy link
Contributor Author

MichelLosier commented Oct 13, 2025

/ci
@MichelLosier MichelLosier marked this pull request as ready for review October 13, 2025 16:20
@MichelLosier MichelLosier requested a review from a team as a code owner October 13, 2025 16:21
@MichelLosier MichelLosier added release_note:fix backport:all-open Backport to all branches that could still receive a release labels Oct 13, 2025
@botelastic botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Oct 13, 2025
@elasticmachine
Copy link
Contributor

elasticmachine commented Oct 13, 2025

Pinging @elastic/fleet (Team:Fleet)
@elasticmachine
Copy link
Contributor

elasticmachine commented Oct 13, 2025

💚 Build Succeeded

Metrics [docs]

✅ unchanged
@MichelLosier MichelLosier self-assigned this Oct 13, 2025
juliaElastic
juliaElastic approved these changes Oct 15, 2025
View reviewed changes
Copy link
Contributor

@juliaElastic juliaElastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@MichelLosier MichelLosier merged commit 1c86d1b into elastic:main Oct 15, 2025
14 checks passed
@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 15, 2025

Starting backport for target branches: 8.18, 8.19, 9.1, 9.2

https://github.com/elastic/kibana/actions/runs/18535604265
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Oct 15, 2025
@MichelLosier
Support var removals on integration upgrade (elastic#238542)
f5e1eb5 
Resolves: elastic#235242

* On package policy upgrade:
  * Adds support for checking global var additions on package upgrade
* Removes stale variables (global, input, or stream) from the original
package policy that are no longer defined in the new package version.

(cherry picked from commit 1c86d1b)
@kibanamachine kibanamachine mentioned this pull request Oct 15, 2025
Merged
@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 15, 2025

💔 Some backports could not be created

Status Branch Result
❌ 8.18 Backport failed because of merge conflicts
❌ 8.19 Backport failed because of merge conflicts
❌ 9.1 Backport failed because of merge conflicts
✅ 9.2

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 238542

Questions ?

Please refer to the Backport tool documentation
@MichelLosier MichelLosier mentioned this pull request Oct 15, 2025
Merged
MichelLosier added a commit to MichelLosier/kibana that referenced this pull request Oct 15, 2025
@MichelLosier
Support var removals on integration upgrade (elastic#238542)
1cf1981 
Resolves: elastic#235242

* On package policy upgrade:
  * Adds support for checking global var additions on package upgrade
* Removes stale variables (global, input, or stream) from the original
package policy that are no longer defined in the new package version.

(cherry picked from commit 1c86d1b)

# Conflicts:
#	x-pack/platform/plugins/shared/fleet/server/services/package_policy.ts
@MichelLosier MichelLosier mentioned this pull request Oct 15, 2025
Closed
MichelLosier added a commit to MichelLosier/kibana that referenced this pull request Oct 15, 2025
@MichelLosier
Support var removals on integration upgrade (elastic#238542)
9f503b7 
Resolves: elastic#235242

* On package policy upgrade:
  * Adds support for checking global var additions on package upgrade
* Removes stale variables (global, input, or stream) from the original
package policy that are no longer defined in the new package version.

(cherry picked from commit 1c86d1b)

# Conflicts:
#	x-pack/platform/plugins/shared/fleet/server/services/package_policy.ts
@MichelLosier MichelLosier mentioned this pull request Oct 15, 2025
Merged
MichelLosier added a commit to MichelLosier/kibana that referenced this pull request Oct 15, 2025
@MichelLosier
Support var removals on integration upgrade (elastic#238542)
c329a7a 
Resolves: elastic#235242

* On package policy upgrade:
  * Adds support for checking global var additions on package upgrade
* Removes stale variables (global, input, or stream) from the original
package policy that are no longer defined in the new package version.

(cherry picked from commit 1c86d1b)

# Conflicts:
#	x-pack/platform/plugins/shared/fleet/server/services/package_policy.ts
@MichelLosier MichelLosier mentioned this pull request Oct 15, 2025
Closed
@MichelLosier
Copy link
Contributor Author

MichelLosier commented Oct 15, 2025

💚 All backports created successfully

Status Branch Result
✅ 9.1
✅ 9.0
✅ 8.19
✅ 8.18

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation
MichelLosier added a commit to MichelLosier/kibana that referenced this pull request Oct 15, 2025
@MichelLosier
Support var removals on integration upgrade (elastic#238542)
e46ec66 
Resolves: elastic#235242

* On package policy upgrade:
  * Adds support for checking global var additions on package upgrade
* Removes stale variables (global, input, or stream) from the original
package policy that are no longer defined in the new package version.

(cherry picked from commit 1c86d1b)

# Conflicts:
#	x-pack/platform/plugins/shared/fleet/server/services/package_policy.ts
MichelLosier added a commit that referenced this pull request Oct 15, 2025
@MichelLosier
[9.1] Support var removals on integration upgrade (#238542) (#239213)
07444b6 
# Backport

This will backport the following commits from `main` to `9.1`:
- [Support var removals on integration upgrade
(#238542)](#238542)

<!--- Backport version: 10.1.0 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Michel
Losier","email":"michel.losier@elastic.co"},"sourceCommit":{"committedDate":"2025-10-15T16:21:04Z","message":"Support
var removals on integration upgrade (#238542)\n\nResolves:
https://github.com/elastic/kibana/issues/235242\n\n* On package policy
upgrade:\n * Adds support for checking global var additions on package
upgrade\n* Removes stale variables (global, input, or stream) from the
original\npackage policy that are no longer defined in the new package
version.","sha":"1c86d1b098457bebfbf008da7a9da781306bfe07","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:all-open","v9.3.0"],"title":"Support
var removals on integration
upgrade","number":238542,"url":"https://github.com/elastic/kibana/pull/238542","mergeCommit":{"message":"Support
var removals on integration upgrade (#238542)\n\nResolves:
https://github.com/elastic/kibana/issues/235242\n\n* On package policy
upgrade:\n * Adds support for checking global var additions on package
upgrade\n* Removes stale variables (global, input, or stream) from the
original\npackage policy that are no longer defined in the new package
version.","sha":"1c86d1b098457bebfbf008da7a9da781306bfe07"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/238542","number":238542,"mergeCommit":{"message":"Support
var removals on integration upgrade (#238542)\n\nResolves:
https://github.com/elastic/kibana/issues/235242\n\n* On package policy
upgrade:\n * Adds support for checking global var additions on package
upgrade\n* Removes stale variables (global, input, or stream) from the
original\npackage policy that are no longer defined in the new package
version.","sha":"1c86d1b098457bebfbf008da7a9da781306bfe07"}},{"url":"https://github.com/elastic/kibana/pull/239189","number":239189,"branch":"9.2","state":"OPEN"}]}]
BACKPORT-->
MichelLosier added a commit that referenced this pull request Oct 15, 2025
@MichelLosier
[8.19] Support var removals on integration upgrade (#238542) (#239217)
1fb7afd 
# Backport

This will backport the following commits from `main` to `8.19`:
- [Support var removals on integration upgrade
(#238542)](#238542)

<!--- Backport version: 10.1.0 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Michel
Losier","email":"michel.losier@elastic.co"},"sourceCommit":{"committedDate":"2025-10-15T16:21:04Z","message":"Support
var removals on integration upgrade (#238542)\n\nResolves:
https://github.com/elastic/kibana/issues/235242\n\n* On package policy
upgrade:\n * Adds support for checking global var additions on package
upgrade\n* Removes stale variables (global, input, or stream) from the
original\npackage policy that are no longer defined in the new package
version.","sha":"1c86d1b098457bebfbf008da7a9da781306bfe07","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:all-open","v9.3.0"],"title":"Support
var removals on integration
upgrade","number":238542,"url":"https://github.com/elastic/kibana/pull/238542","mergeCommit":{"message":"Support
var removals on integration upgrade (#238542)\n\nResolves:
https://github.com/elastic/kibana/issues/235242\n\n* On package policy
upgrade:\n * Adds support for checking global var additions on package
upgrade\n* Removes stale variables (global, input, or stream) from the
original\npackage policy that are no longer defined in the new package
version.","sha":"1c86d1b098457bebfbf008da7a9da781306bfe07"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/238542","number":238542,"mergeCommit":{"message":"Support
var removals on integration upgrade (#238542)\n\nResolves:
https://github.com/elastic/kibana/issues/235242\n\n* On package policy
upgrade:\n * Adds support for checking global var additions on package
upgrade\n* Removes stale variables (global, input, or stream) from the
original\npackage policy that are no longer defined in the new package
version.","sha":"1c86d1b098457bebfbf008da7a9da781306bfe07"}},{"url":"https://github.com/elastic/kibana/pull/239189","number":239189,"branch":"9.2","state":"OPEN"}]}]
BACKPORT-->
@kibanamachine kibanamachine added v8.19.6 backport missing Added to PRs automatically when the are determined to be missing a backport. labels Oct 15, 2025
@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 16, 2025

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @MichelLosier
mgadewoll pushed a commit to tkajtoch/kibana that referenced this pull request Oct 17, 2025
@MichelLosier @mgadewoll
Support var removals on integration upgrade (elastic#238542)
efd004a 
Resolves: elastic#235242

* On package policy upgrade:
  * Adds support for checking global var additions on package upgrade
* Removes stale variables (global, input, or stream) from the original
package policy that are no longer defined in the new package version.
@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 17, 2025

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @MichelLosier
@MichelLosier MichelLosier mentioned this pull request Oct 20, 2025
Open
@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 20, 2025

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @MichelLosier
1 similar comment
@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 21, 2025

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @MichelLosier
@jen-huang jen-huang added backport:version Backport to applied version labels and removed backport:all-open Backport to all branches that could still receive a release labels Oct 21, 2025
@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 21, 2025

Starting backport for target branches: 8.19, 9.1

https://github.com/elastic/kibana/actions/runs/18700409732
@jen-huang
Copy link
Contributor

jen-huang commented Oct 21, 2025

Starting backport for target branches: 8.19, 9.1

Trying to cancel the above run since these backports are already merged. Might need a backport to 9.2 though.
kibanamachine added a commit that referenced this pull request Oct 22, 2025
@kibanamachine @MichelLosier
[9.2] Support var removals on integration upgrade (#238542) (#239189)
b1a64ab 
# Backport

This will backport the following commits from `main` to `9.2`:
- [Support var removals on integration upgrade
(#238542)](#238542)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Michel
Losier","email":"michel.losier@elastic.co"},"sourceCommit":{"committedDate":"2025-10-15T16:21:04Z","message":"Support
var removals on integration upgrade (#238542)\n\nResolves:
https://github.com/elastic/kibana/issues/235242\n\n* On package policy
upgrade:\n * Adds support for checking global var additions on package
upgrade\n* Removes stale variables (global, input, or stream) from the
original\npackage policy that are no longer defined in the new package
version.","sha":"1c86d1b098457bebfbf008da7a9da781306bfe07","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:all-open","v9.3.0"],"title":"Support
var removals on integration
upgrade","number":238542,"url":"https://github.com/elastic/kibana/pull/238542","mergeCommit":{"message":"Support
var removals on integration upgrade (#238542)\n\nResolves:
https://github.com/elastic/kibana/issues/235242\n\n* On package policy
upgrade:\n * Adds support for checking global var additions on package
upgrade\n* Removes stale variables (global, input, or stream) from the
original\npackage policy that are no longer defined in the new package
version.","sha":"1c86d1b098457bebfbf008da7a9da781306bfe07"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/238542","number":238542,"mergeCommit":{"message":"Support
var removals on integration upgrade (#238542)\n\nResolves:
https://github.com/elastic/kibana/issues/235242\n\n* On package policy
upgrade:\n * Adds support for checking global var additions on package
upgrade\n* Removes stale variables (global, input, or stream) from the
original\npackage policy that are no longer defined in the new package
version.","sha":"1c86d1b098457bebfbf008da7a9da781306bfe07"}}]}]
BACKPORT-->

Co-authored-by: Michel Losier <michel.losier@elastic.co>
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Oct 22, 2025
nickpeihl pushed a commit to nickpeihl/kibana that referenced this pull request Oct 23, 2025
@MichelLosier @nickpeihl
Support var removals on integration upgrade (elastic#238542)
68e20f3 
Resolves: elastic#235242

* On package policy upgrade:
  * Adds support for checking global var additions on package upgrade
* Removes stale variables (global, input, or stream) from the original
package policy that are no longer defined in the new package version.
NicholasPeretti pushed a commit to NicholasPeretti/kibana that referenced this pull request Oct 27, 2025
@MichelLosier @NicholasPeretti
Support var removals on integration upgrade (elastic#238542)
f46eb23 
Resolves: elastic#235242

* On package policy upgrade:
  * Adds support for checking global var additions on package upgrade
* Removes stale variables (global, input, or stream) from the original
package policy that are no longer defined in the new package version.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:version Backport to applied version labels release_note:fix Team:Fleet Team label for Observability Data Collection Fleet team v8.19.6 v9.1.6 v9.2.0 v9.3.0

5 participants

@MichelLosier @elasticmachine @kibanamachine @jen-huang @juliaElastic