Cleanup available rollbacks by pchila · Pull Request #11562 · elastic/elastic-agent

pchila · 2025-12-03T15:19:01Z

What does this PR do?

This PR will cleanup available rollbacks when:

initiating a new upgrade, to avoid increasing the disk space needed to 3x the size of an agent installation
when an available upgrade expires by running a goroutine that will periodically run a check and cleanup.

Why is it important?

To avoid having to clean up manually when upgrading again an agent still within the rollback window and not to wait until the agent restart to clean up obsolete installs

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~[ ] I have added an entry in ./changelog/fragments using the changelog tool~~
~~[ ] I have added an integration test or an E2E test~~

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

How are we going to support this in production?
How are we going to measure its adoption?
How are we going to debug this?
What are the metrics I should take care of?
...

mergify · 2025-12-03T15:19:36Z

This pull request does not have a backport label. Could you fix it @pchila? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

elasticmachine · 2025-12-11T12:31:57Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

blakerouse

This overall looks good, but the part that really prevents me from giving this a +1 is an integration test. I think having this in an integration tests is critical. We should full observe in the test that the available rollback is removed once the upgrade is complete.

internal/pkg/agent/application/upgrade/manual_rollback.go

internal/pkg/agent/cmd/run.go

testing/integration/ess/upgrade_rollback_test.go

pchila · 2025-12-12T18:08:05Z

This overall looks good, but the part that really prevents me from giving this a +1 is an integration test. I think having this in an integration tests is critical. We should full observe in the test that the available rollback is removed once the upgrade is complete.

@blakerouse have a look at 3d6063a and fea6eb1

…pired_rollback_after_upgrading_to_a_repackaged_version on windows

cmacknz

Generally looks good after latest changes, just need CI to pass.

testing/integration/ess/upgrade_rollback_test.go

…lear_expired_rollback_after_upgrading_to_a_repackaged_version on windows

…on tests

elasticmachine · 2026-01-08T09:13:05Z

💔 Build Failed

Buildkite Build
Commit: fd7a5e6

Failed CI Steps

History

💔 Build #33017 failed 8ae737a
💔 Build #32914 failed 8279d1b
💔 Build #32885 failed 0966eaa
💔 Build #32769 failed 5f92818

cc @pchila

* Cleanup rollbacks when triggering a new upgrade * refactor available rollback normalization at startup * WIP - scheduled rollback cleanup * WIP - wire scheduled rollback cleanup at agent startup * remove appDone handling from PeriodicallyCleanRollbacks * Pass the correct relative path to the rollback cleanup goroutine * refactor from commit and repackaged fixtures for rollback tests * Add Hash() to agent integration test fixture * Define a minimum cleanup interval for available rollbacks * create ttl marker files without world-readable permissions * introduce integration test for automatic cleanup of expired rollbacks * fixup! Define a minimum cleanup interval for available rollbacks * fixup! fixup! Define a minimum cleanup interval for available rollbacks * fixup! Add Hash() to agent integration test fixture * Add cleanup rollback test for multiple upgrades within the window * Use an additional subcontext for cleanup goroutine * add debug logging and skip TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version on windows * fixup! add debug logging and skip TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version on windows * Check upgrade details status more frequently during upgrade integration tests --------- Co-authored-by: Eric Beahan <eric.beahan@elastic.co> (cherry picked from commit 5e3c865)

* Cleanup rollbacks when triggering a new upgrade * refactor available rollback normalization at startup * WIP - scheduled rollback cleanup * WIP - wire scheduled rollback cleanup at agent startup * remove appDone handling from PeriodicallyCleanRollbacks * Pass the correct relative path to the rollback cleanup goroutine * refactor from commit and repackaged fixtures for rollback tests * Add Hash() to agent integration test fixture * Define a minimum cleanup interval for available rollbacks * create ttl marker files without world-readable permissions * introduce integration test for automatic cleanup of expired rollbacks * fixup! Define a minimum cleanup interval for available rollbacks * fixup! fixup! Define a minimum cleanup interval for available rollbacks * fixup! Add Hash() to agent integration test fixture * Add cleanup rollback test for multiple upgrades within the window * Use an additional subcontext for cleanup goroutine * add debug logging and skip TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version on windows * fixup! add debug logging and skip TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version on windows * Check upgrade details status more frequently during upgrade integration tests --------- (cherry picked from commit 5e3c865) Co-authored-by: Paolo Chilà <paolo.chila@elastic.co> Co-authored-by: Eric Beahan <eric.beahan@elastic.co>

* Cleanup rollbacks when triggering a new upgrade * refactor available rollback normalization at startup * WIP - scheduled rollback cleanup * WIP - wire scheduled rollback cleanup at agent startup * remove appDone handling from PeriodicallyCleanRollbacks * Pass the correct relative path to the rollback cleanup goroutine * refactor from commit and repackaged fixtures for rollback tests * Add Hash() to agent integration test fixture * Define a minimum cleanup interval for available rollbacks * create ttl marker files without world-readable permissions * introduce integration test for automatic cleanup of expired rollbacks * fixup! Define a minimum cleanup interval for available rollbacks * fixup! fixup! Define a minimum cleanup interval for available rollbacks * fixup! Add Hash() to agent integration test fixture * Add cleanup rollback test for multiple upgrades within the window * Use an additional subcontext for cleanup goroutine * add debug logging and skip TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version on windows * fixup! add debug logging and skip TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version on windows * Check upgrade details status more frequently during upgrade integration tests --------- Co-authored-by: Eric Beahan <eric.beahan@elastic.co>

pchila self-assigned this Dec 3, 2025

pchila added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team Team:Elastic-Agent Label for the Agent team labels Dec 3, 2025

pchila added backport-skip skip-changelog labels Dec 3, 2025

pchila force-pushed the cleanup-rollbacks branch from 4d4ccd2 to 3ac0d8a Compare December 11, 2025 09:41

pchila added the enhancement New feature or request label Dec 11, 2025

pchila marked this pull request as ready for review December 11, 2025 12:31

pchila requested a review from a team as a code owner December 11, 2025 12:31

pchila requested review from blakerouse and swiatekm December 11, 2025 12:31

blakerouse reviewed Dec 11, 2025

View reviewed changes

internal/pkg/agent/application/upgrade/manual_rollback.go Outdated Show resolved Hide resolved

cmacknz reviewed Dec 11, 2025

View reviewed changes

internal/pkg/agent/cmd/run.go Outdated Show resolved Hide resolved

pchila added 11 commits December 12, 2025 15:28

Cleanup rollbacks when triggering a new upgrade

5944572

refactor available rollback normalization at startup

abb57f8

WIP - scheduled rollback cleanup

762c7b1

WIP - wire scheduled rollback cleanup at agent startup

77ba5c1

remove appDone handling from PeriodicallyCleanRollbacks

18eb035

Pass the correct relative path to the rollback cleanup goroutine

87de1ce

refactor from commit and repackaged fixtures for rollback tests

6720e81

Add Hash() to agent integration test fixture

6ac1bd7

Define a minimum cleanup interval for available rollbacks

f6867fe

create ttl marker files without world-readable permissions

589b42a

introduce integration test for automatic cleanup of expired rollbacks

3d6063a

pchila force-pushed the cleanup-rollbacks branch from bee1173 to 3d6063a Compare December 12, 2025 14:29

fixup! Define a minimum cleanup interval for available rollbacks

d447b9e

cmacknz reviewed Dec 12, 2025

View reviewed changes

testing/integration/ess/upgrade_rollback_test.go Show resolved Hide resolved

fixup! fixup! Define a minimum cleanup interval for available rollbacks

6e1a96c

pchila added 2 commits December 12, 2025 18:11

fixup! Add Hash() to agent integration test fixture

d918a5c

Add cleanup rollback test for multiple upgrades within the window

fea6eb1

pchila and others added 3 commits December 12, 2025 19:11

Use an additional subcontext for cleanup goroutine

4bb9432

add debug logging and skip TestCleanupRollbacks/agent_should_clear_ex…

30f8d3e

…pired_rollback_after_upgrading_to_a_repackaged_version on windows

Merge branch 'main' into cleanup-rollbacks

c89881c

This was referenced Dec 19, 2025

Allow elastic-agent to start a new upgrade by cleaning up the available rollbacks #6892

Closed

Wait for elastic-agent watcher process to complete at startup and schedule cleanup #6882

Closed

Set manual rollback default window to 7 days #11955

Merged

cmacknz reviewed Dec 29, 2025

View reviewed changes

testing/integration/ess/upgrade_rollback_test.go Outdated Show resolved Hide resolved

fixup! add debug logging and skip TestCleanupRollbacks/agent_should_c…

5f92818

…lear_expired_rollback_after_upgrading_to_a_repackaged_version on windows

ebeahan added backport-9.3 Automated backport to the 9.3 branch and removed backport-skip labels Jan 5, 2026

ebeahan added 2 commits January 5, 2026 09:35

Merge branch 'main' into cleanup-rollbacks

0966eaa

Merge branch 'main' into cleanup-rollbacks

8279d1b

ebeahan mentioned this pull request Jan 5, 2026

Add rollback_window to elastic agent configuration with a default value of 7d #6881

Closed

pchila added 2 commits January 7, 2026 08:46

Merge branch 'main' into cleanup-rollbacks

8ae737a

Check upgrade details status more frequently during upgrade integrati…

fd7a5e6

…on tests

Merge branch 'main' into cleanup-rollbacks

40d3c25

cmacknz approved these changes Jan 12, 2026

View reviewed changes

ebeahan merged commit 5e3c865 into elastic:main Jan 12, 2026
22 checks passed

mergify bot mentioned this pull request Jan 12, 2026

[9.3] (backport #11562) Cleanup available rollbacks #12208

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cleanup available rollbacks#11562

Cleanup available rollbacks#11562
ebeahan merged 24 commits intoelastic:mainfrom
pchila:cleanup-rollbacks

pchila commented Dec 3, 2025 •

edited

Loading

mergify bot commented Dec 3, 2025

elasticmachine commented Dec 11, 2025

blakerouse left a comment

Uh oh!

Uh oh!

Uh oh!

pchila commented Dec 12, 2025

cmacknz left a comment

Uh oh!

elasticmachine commented Jan 8, 2026 •

edited

Loading

Uh oh!

Labels

5 participants

Conversation

pchila commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

mergify bot commented Dec 3, 2025

elasticmachine commented Dec 11, 2025

blakerouse left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pchila commented Dec 12, 2025

cmacknz left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

Labels

5 participants

pchila commented Dec 3, 2025 •

edited

Loading

elasticmachine commented Jan 8, 2026 •

edited

Loading