Skip to content

[Fleet] Update minimun package spec version to 2.3#214600

Merged
jsoriano merged 20 commits intoelastic:mainfrom
jsoriano:update-spec-min
Mar 25, 2025
Merged

[Fleet] Update minimun package spec version to 2.3#214600
jsoriano merged 20 commits intoelastic:mainfrom
jsoriano:update-spec-min

Conversation

@jsoriano
Copy link
Member

@jsoriano jsoriano commented Mar 14, 2025

Summary

Update minimum package spec version to 2.3.

This effectively removes availability of some problematic packages with lower format versions in 9.x, when using the default configuration.

FTR, serverless already uses a minimum spec version of 3.0.

This affects the following packages from the integrations repository, when using kibana.version=9.0:

If no kibana.version constraint is used, it affects the following packages:

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

  • Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
  • Documentation was added for features that require explanation or tutorials
  • Unit or functional tests were updated or added to match the most common scenarios
  • If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
  • This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
  • Flaky Test Runner was used on any tests changed
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Identify risks

  • Late introduction of changes in availability of packages. Though this can be considered a fix as it removes packages that are not expected to be available in 9.x stacks.
@jsoriano jsoriano self-assigned this Mar 14, 2025
@jsoriano jsoriano requested a review from a team as a code owner March 14, 2025 15:44
@botelastic botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Mar 14, 2025
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@jsoriano jsoriano requested a review from a team as a code owner March 17, 2025 22:38
Copy link
Contributor

@dmlemeshko dmlemeshko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we update FTR server configuration, I assume we need to do the same for Scout?

@jsoriano
Copy link
Member Author

I am struggling to override the config in functional tests.

If I use quotes, like this:

        `--xpack.fleet.internal.registry.spec.min='1.0'`,

URLs are incorrectly built, it looks like quotes end up being part of the url:

 proc [kibana] [2025-03-18T17:32:12.203+01:00][ERROR][plugins.fleet] Failed to fetch latest version of synthetics from registry: '400 Bad Request' error response from package registry at https://epr.elastic.co/search?package=synthetics&prerelease=true&spec.min=%271.0%27&spec.max=3.3 {"service":{"node":{"roles":["background_tasks","ui"]}}}

If I don't use quotes, like this:

        `--xpack.fleet.internal.registry.spec.min=1.0`,

Kibana fails to start with this error:

 proc [kibana]  FATAL  Error: [config validation of [xpack.fleet].internal.registry.spec.min]: expected value of type [string] but got [number]
@kpollich
Copy link
Member

@jsoriano - Could you try with a plain quoted string instead of the template literal e.g.

"--xpack.fleet.internal.registry.spec.min='1.0'",

I don't see why it would matter, but maybe it's worth trying.

I also think the URL escaping might be a bug in Fleet when this value is configured via CLI instead of kibana.yml.

@jsoriano
Copy link
Member Author

I don't see why it would matter, but maybe it's worth trying.

Yes, it does the same.

I also think the URL escaping might be a bug in Fleet when this value is configured via CLI instead of kibana.yml.

But maybe Kibana should not fail if 1.0 is not quoted? It should be possible to use it as string even if it also parses as number 🤔

@MichelLosier
Copy link
Contributor

I see the string validator has a coerceFromNumber option

Does that help when not wrapping 1.0 in quotes?

@jsoriano
Copy link
Member Author

I see the string validator has a coerceFromNumber option

Does that help when not wrapping 1.0 in quotes?

Yes, I was just trying that and it helps, thanks. The problem is that it coerces 1.0 to 1, so we need some formatting later before building the request.

@jsoriano
Copy link
Member Author

Possible fix pushed to 6f41b9b, let's see what CI thinks 🙂

}

function formatSpecVersion(version: string): string {
// Version can be coerced from number when obtained through flags, in these cases X.0 versions are coerced to X.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still a problem for versions whose minor ends with 0, for example 2.1, 2.10 and 2.100 will be parsed as 2.1, and coerced to the 2.1 string.

No idea what we can do to avoid flags being parsed as numbers.

Copy link
Contributor

@MichelLosier MichelLosier Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we foresee problems with that loss of precision in how this value is used?

Seems like if ideally we want to keep this value as a string, one thought is we could enhance the StringType validator to have an isSemver option, much like it has a hostname one.

I see the repo already has the semver library already as a dependency, so we could lean on the semver.valid() util from that for the implementation of that option

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we foresee problems with that loss of precision in how this value is used?

Yes, it is a problem here because we expect <major>.<minor> format. With the "fix" introduced here it will be only a problem with minors that are multiples of 10.

Seems like if ideally we want to keep this value as a string, one thought is we could enhance the StringType validator to have an isSemver option, much like it has a hostname one.

I have tried some things but it looks like the value is directly parsed as number, so there is some loss.

Maybe it is expected for strings to be used as quotes, though it didn't work well when used in tests.

We can investigate this further in the future, this PR at least doesn't make things worse 🙂

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue created for further investigation: #215251

Copy link
Contributor

@MichelLosier MichelLosier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just a suggestion on how we may avoid the lossy string -> number -> string conversion

}

function formatSpecVersion(version: string): string {
// Version can be coerced from number when obtained through flags, in these cases X.0 versions are coerced to X.
Copy link
Contributor

@MichelLosier MichelLosier Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we foresee problems with that loss of precision in how this value is used?

Seems like if ideally we want to keep this value as a string, one thought is we could enhance the StringType validator to have an isSemver option, much like it has a hostname one.

I see the repo already has the semver library already as a dependency, so we could lean on the semver.valid() util from that for the implementation of that option

registry: {
kibanaVersionCheckEnabled: false,
spec: {
min: '1.0',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano the goal of that test what to mimic the cloud config, I am worried that if we introduce a change here, we may accidentally break cloud, it is because there no apm package compatible? should it be fixed by publishing one instead?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APM packages are not released as GA in the registry since 8.4.2. So not sure if we can assume that there is going to be a package there.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could it be that the GA package is bundled in GA Kibana, but the one in the registry is used for development? 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I think in development we are using the old APM package

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created issue about this: #215400

@jsoriano
Copy link
Member Author

If we update FTR server configuration, I assume we need to do the same for Scout?

@dmlemeshko the changes in the FTR configurations are intended to workaround #215400, I wouldn't modify other testing configurations if not needed.

Let me know if you were thinking in other modifications.

@jsoriano
Copy link
Member Author

jsoriano commented Mar 24, 2025

Pushed a change to remove the flag overrides from x-pack/test/functional/config.base.js, to see if we can add these overrides only to tests that use the registry [to install APM package].

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

cc @jsoriano

@dmlemeshko dmlemeshko self-requested a review March 25, 2025 12:03
Copy link
Contributor

@dmlemeshko dmlemeshko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jsoriano jsoriano merged commit 8d81ed4 into elastic:main Mar 25, 2025
10 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 9.0

https://github.com/elastic/kibana/actions/runs/14059131231

@kibanamachine
Copy link
Contributor

💔 Backport failed

The pull request could not be backported due to the following error:
Git clone failed with exit code: 128

Manual backport

To create the backport manually run:

node scripts/backport --pr 214600

Questions ?

Please refer to the Backport tool documentation

@kibanamachine
Copy link
Contributor

Starting backport for target branches: 9.0

https://github.com/elastic/kibana/actions/runs/14059131231

@jsoriano
Copy link
Member Author

💚 All backports created successfully

Status Branch Result
9.0

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

jsoriano added a commit to jsoriano/kibana that referenced this pull request Mar 25, 2025
Update minimum package spec version to 2.3.

This effectively removes availability of some problematic packages with
lower format versions in 9.x, when using the default configuration.

Serverless already uses a minimum spec version of 3.0.

This affects a small set of packages from the integrations repository,
for which there are resolution plans.

(cherry picked from commit 8d81ed4)
jsoriano added a commit that referenced this pull request Mar 25, 2025
…215858)

# Backport

This will backport the following commits from `main` to `9.0`:
- [[Fleet] Update minimun package spec version to 2.3
(#214600)](#214600)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Jaime Soriano
Pastor","email":"jaime.soriano@elastic.co"},"sourceCommit":{"committedDate":"2025-03-25T12:04:29Z","message":"[Fleet]
Update minimun package spec version to 2.3 (#214600)\n\nUpdate minimum
package spec version to 2.3.\n\nThis effectively removes availability of
some problematic packages with\nlower format versions in 9.x, when using
the default configuration.\n\nServerless already uses a minimum spec
version of 3.0.\n\nThis affects a small set of packages from the
integrations repository,\nfor which there are resolution
plans.","sha":"8d81ed4a5b6f540ce9847bdd0477f52d514031b3","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:prev-minor","v9.1.0"],"title":"[Fleet]
Update minimun package spec version to
2.3","number":214600,"url":"https://github.com/elastic/kibana/pull/214600","mergeCommit":{"message":"[Fleet]
Update minimun package spec version to 2.3 (#214600)\n\nUpdate minimum
package spec version to 2.3.\n\nThis effectively removes availability of
some problematic packages with\nlower format versions in 9.x, when using
the default configuration.\n\nServerless already uses a minimum spec
version of 3.0.\n\nThis affects a small set of packages from the
integrations repository,\nfor which there are resolution
plans.","sha":"8d81ed4a5b6f540ce9847bdd0477f52d514031b3"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/214600","number":214600,"mergeCommit":{"message":"[Fleet]
Update minimun package spec version to 2.3 (#214600)\n\nUpdate minimum
package spec version to 2.3.\n\nThis effectively removes availability of
some problematic packages with\nlower format versions in 9.x, when using
the default configuration.\n\nServerless already uses a minimum spec
version of 3.0.\n\nThis affects a small set of packages from the
integrations repository,\nfor which there are resolution
plans.","sha":"8d81ed4a5b6f540ce9847bdd0477f52d514031b3"}}]}]
BACKPORT-->
cqliu1 pushed a commit to cqliu1/kibana that referenced this pull request Mar 31, 2025
Update minimum package spec version to 2.3.

This effectively removes availability of some problematic packages with
lower format versions in 9.x, when using the default configuration.

Serverless already uses a minimum spec version of 3.0.

This affects a small set of packages from the integrations repository,
for which there are resolution plans.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release_note:fix Team:Fleet Team label for Observability Data Collection Fleet team v9.0.0 v9.1.0

8 participants