[Fleet] Add package rollback API

[jillguyonnet]

Summary

Closes https://github.com/elastic/ingest-dev/issues/5445

This PR adds a public API endpoint POST kbn:/api/fleet/epm/packages/{packageName}/rollback that rolls back an integration to the previous installed version, provided that previous version had been saved and that all integration policies (in all spaces) have corresponding saved previous revisions.

This endpoint updates integration policies across all spaces. For example, given the following:

• The system integration was installed on 2.2.0 with integration policies (also on 2.2.0) in multiple spaces.
• The integration was upgraded to 2.3.2 and all integration policies were also upgraded.
  Then rolling back the integration will install 2.2.0 and update integration policies in all spaces to the latest revision they were on for 2.2.0.

Rollback will fail if:

• The integration has no previous_version property set (set on upgrade, see [Fleet] Save package policy previous revision on package upgrade #222779).
• At least one integration policy (in any space) doesn't have a saved previous revision (SO with id {id}:prev).
• At least one integration policy (in any space) has a saved previous revision with a different previous version.

Implementation details

1. Package policies rollback step 1
   1. Create temporary SO copies in order to reverse the rollback in case of failure.
   2. Update SO with data from saved previous revisions.
2. Package rollback
3. Package policies rollback step 2
   • If package rollback succeeded, delete temporary SO copies and previous revision SO + bump agent policy revisions.
   • If package rollback failed, restore SO to pre-rollback (using temporary SO copies) and delete these copies.

Testing

1. Create a custom space with default accesses.
2. In the default space install a package on an older version and make sure to have a package policy (edit the version in the browser URL and add the package). Choose a version old enough to have a few more recent versions to test with.
3. In the custom space, add the package on the same older version in order to create a package policy. You should now have a package policy on this older version on both spaces.
4. Go to Dev Tools and issue a rollback request: it should fail with No previous version found for package {pkgName}.
5. Force upgrade the package to a newer (not the latest) version with

   POST kbn:/api/fleet/epm/packages/{pkgName}/{pkgVersion}
   {
     "force": true
   }
   

   This doesn't upgrade the package policies.
6. Issue a rollback request: it should fail with No previous version found for package policies: [policyId1, policyId2].
7. Force upgrade the package again to a newer version.
8. In the UI, upgrade both package policies.
9. Go to Dev Tools and issue a rollback request: it should fail with Wrong previous version for package policies: [policyId1, policyId2]. This is because the package was upgraded from v1 to v2 to v3, but policies were directly upgraded from v1 to v3.
10. Force upgrade the package again to a newer version.
11. In the UI, upgrade both package policies.
12. Take note of the revisions the agent policies are on.
13. Go to Dev Tools and issue a rollback request: this time, it should succeed and roll back the package and its policies in both spaces to the previous version.
14. Check in the UI that the rollback looks correct (agent policy revisions should have been bumped). For completeness, you can check the saved objects of the package and the package policies:

• the package SO should have the correct (older) version and previous_version: null
• the package policy SO storing the revision with the previous version (with id ending in :prev) should have been deleted
• the current package policy SO should have the correct (older) version, latest_revision: true and revision number bumped by 1
• there should be no temporary SO (with id ending in :copy)

Checklist

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
• Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Risk of incorrect integration policy data after rolling back the integration to its previous version.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

It would be great to add integration tests for the new rollback API and do manual testing with agents using the package policies and ingesting data to data streams, to make sure the rollback doesn't disrupt data ingestion.

[criamico]

Adding to this, it would be good to test with input packages as well to make sure it doesn't break them.

[criamico]

Adding to this, it would be good to test with input packages as well to make sure it doesn't break them.

[jillguyonnet]

@elasticmachine merge upstream

[jillguyonnet]

@juliaElastic @criamico thank you for your feedback. I'm still working on functional tests, but I pushed a bunch of improvements and updated the PR description with additional test cases. Please feel free to try and break this. 🙂

[jillguyonnet]

@elasticmachine merge upstream

[kibanamachine]

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#8644

[❌] x-pack/test/fleet_api_integration/config.epm.ts: 75/100 tests passed.

see run history

[juliaElastic]

LGTM

[jillguyonnet]

@elasticmachine merge upstream

[jillguyonnet]

Thanks for the review @juliaElastic - FYI I had to increase the retries and timeout in this test as the extra logic caused it to be too slow. It made me realise that we don't exclude rollback logic from managed policies, which should perhaps be addressed as a followup, WDYT?

[juliaElastic]

Why is setup trying to run rollback?

I agree that we probably shouldn't allow rolling back managed policies.

[jillguyonnet]

Why is setup trying to run rollback?

It isn't - it's the saving previous revisions logic here that adds extra time. Not sure why this didn't come up in the original PR.
Edit: I've kicked off a flaky test runner build to check if it's robust enough with these settings.

I agree that we probably shouldn't allow rolling back managed policies.

OK if I open a followup issue for this?

[juliaElastic]

I see, it makes sense. A follow up issue sounds good.

[jillguyonnet]

@elastic/core-docs @elastic/experience-docs Could I please get your review on this?

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#8669

[✅] x-pack/platform/test/fleet_api_integration/config.epm.ts: 100/100 tests passed.

see run history

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#8679

[✅] x-pack/platform/test/fleet_api_integration/config.fleet.ts: 100/100 tests passed.

see run history

[florent-leborgne]

Hi! Looks like the API docs output files are no longer part of this PR. In case you'll re-add them, I mainly have one comment about making sure that the API's version availability is correctly set so it appears clearly in the docs:

If there is a version requirement to use this API, it should be specified with an availability attribute like this I believe (see example):

options: {
        availability: {
          since: '9.1.0',
        },

[florent-leborgne]

Thanks for specifying the availability options 🙏 LGTM for docs

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 609dd24

Failed CI Steps

• Fleet Cypress Tests #2

Test Failures

• [job] [logs] Fleet Cypress Tests #2 / View agents list Agent status filter should filter on healthy and unhealthy
• [job] [logs] Fleet Cypress Tests #2 / View agents list Bulk actions should allow to bulk upgrade agents and cancel that upgrade

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	1438	1455	+17

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
fleet	98	101	+3

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	167.0KB	167.1KB	+43.0B

Unknown metric groups

API count

id	before	after	diff
fleet	1569	1586	+17

History

• 💔 Build #320809 failed 9f3d312
• 💛 Build #320678 was flaky 498eeb3
• 💚 Build #319759 succeeded 51dabad
• 💔 Build #319288 failed f84b16c
• 💔 Build #319194 failed 7c71bd5
• 💔 Build #318894 failed edefeb0

cc @jillguyonnet