[deb/rpm] copy run folder from old agent installation

[pkoutsovasilis]

What does this PR do?

This PR improves the upgrade experience of Elastic Agent installed via DEB and RPM packages. It updates the pre-installation script (preinstall.sh.tmpl) to copy the run directory from the previous agent installation into the new version's folder during an upgrade, preserving agent state that would otherwise be lost.

Additionally, it extends integration tests to:

• Introduce validation to ensure the run directory and migration marker are properly copied across an upgrade.
• Add Debian OS support in DEB-related upgrade tests.

A new helper function FindRunDir is also introduced in the testing framework to locate the agent's run directory based on versioned data paths.

Based on my investigation:

1. Fleet-related configuration is at /etc/elastic-agent/fleet.enc which is preserved across upgrades
2. Elastic-agent config file is at /etc/elastic-agent/elastic-agent.yml which is preserved across upgrades
3. Legacy state.yml and state.enc are getting copied during upgrades
4. With this PR, run folder which contains components registry is also copied during ugprades

Why is it important?

Currently, when an Elastic Agent installed via a DEB or RPM package is upgraded using a package manager (apt/rpm), it does not preserve the previous state saved under run folder resulting in a poor upgrade experience. This PR ensures that the run directory is correctly migrated, matching the behaviour expected during upgrades.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

No disruptive impact expected. This change improves the upgrade process without altering current workflows for users. Upgrades will now correctly migrate the agent’s runtime state.

How to test this PR locally

• Install an Elastic Agent via a DEB package on Ubuntu or Debian.
• Enroll and start the agent.
• Place a marker file inside the agent's run directory.
• Upgrade the agent using apt install or dpkg -i.
• Verify that the marker file is preserved in the new run directory after upgrade.
• Run the updated integration tests under testing/integration/.

Related issues

• Relates Preserve agent state when upgrading using package manager #3832

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[ycombinator]

Tested this PR locally and it works as expected:

$ dpkg -i elastic-agent-9.0.0-arm64.deb
Selecting previously unselected package elastic-agent.
...
systemd enable/restart elastic-agent
$ touch /var/lib/elastic-agent/data/elastic-agent-9.0.0-9786ac/run/panos_was_here
$ ls -l /var/lib/elastic-agent/data/elastic-agent-9.0.0-9786ac/run/
total 4
drwxr-x--- 1 root root 36 Apr 29 13:56 beat
drwxrwx--- 1 root root 34 Apr 29 13:56 filestream-monitoring
drwxr-x--- 1 root root 36 Apr 29 13:56 http
-rw-r--r-- 1 root root  0 Apr 29 13:56 panos_was_here
drwxr-x--- 1 root root 30 Apr 29 13:56 system
$ dpkg -i elastic-agent-9.1.0-SNAPSHOT-arm64.deb
(Reading database ... 32803 files and directories currently installed.)
...
found /var/lib/elastic-agent/data/elastic-agent-9.0.0-9786ac/run, copy to /var/lib/elastic-agent/data/elastic-agent-9.1.0-SNAPSHOT-5a8714
...
systemd enable/restart elastic-agent
$ ls -l /var/lib/elastic-agent/data/elastic-agent-9.1.0-SNAPSHOT-5a8714/run/
total 0
drwxr-x--- 1 root root 36 Apr 29 13:57 beat
drwxrwx--- 1 root root 34 Apr 29 13:57 filestream-monitoring
drwxr-x--- 1 root root 36 Apr 29 13:57 http
-rw-r--r-- 1 root root  0 Apr 29 13:57 panos_was_here
drwxr-x--- 1 root root 30 Apr 29 13:57 system

Left some minor comments on the code.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 97003d4

Failed CI Steps

• Win2025:sudo:upgrade
• x86_64:sudo: install-uninstall

History

• 💛 Build #20676 was flaky 5a87142
• 💛 Build #20664 was flaky 9d04234
• 💛 Build #20650 was flaky 7272d04
• 💔 Build #20644 failed 7598b06
• 💚 Build #20594 succeeded 1ea23a9

cc @pkoutsovasilis

[github-actions]

@Mergifyio backport 8.17 8.18 8.19 9.0

[mergify]

backport 8.17 8.18 8.19 9.0

✅ Backports have been created

Details

• #8058 [8.17] (backport #7999) [deb/rpm] copy run folder from old agent installation has been created for branch 8.17 but encountered conflicts
• #8059 [8.18] (backport #7999) [deb/rpm] copy run folder from old agent installation has been created for branch 8.18 but encountered conflicts
• #8060 [8.19] (backport #7999) [deb/rpm] copy run folder from old agent installation has been created for branch 8.19 but encountered conflicts
• #8061 [9.0] (backport #7999) [deb/rpm] copy run folder from old agent installation has been created for branch 9.0