Fix signature verification using the upgrade command with the --source-uri flag for fleet-managed agents

[michel-laterman]

What does this PR do?

Fix PGP verification to use the .asc file located in the dir specified by --source-uri when upgrading a fleet-managed agent.

Why is it important?

Upgrading using a local source currently requires a work-around.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

N/A

Related issues

• Closes When using a file:// source URI to upgrade from a local directory, Elastic Agent does not fetch the .asc from the source URI #11152

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: e495cac

Failed CI Steps

• Win2022:sudo:fleet
• Win2022:sudo:install-uninstall
• x86_64:sudo: fleet
• :open-pull-request: :kubernetes: v1.27.16:amd64:complete-wolfi

History

• 💔 Build #32369 failed da13908
• 💔 Build #32300 failed b1d0696
• 💔 Build #32295 failed e8d50e0
• 💔 Build #32286 failed e14d737
• 💔 Build #32281 failed 5ecccfe

cc @michel-laterman

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[ycombinator]

I think you need to play the same game that you have in this PR for the .asc file with the .sha512 file as well. Otherwise, you get the following error, even though the .sha512 file is present in the folder specified by --source-uri:

$ sudo elastic-agent upgrade --force --source-uri file://$(pwd)/development/github/elastic-agent/build/distributions/ 9.4.0-SNAPSHOT
Error: Failed trigger upgrade of daemon: failed verification of agent binary: failed to verify SHA512 hash: could not read checksum file: checksum for "elastic-agent-9.4.0-SNAPSHOT-linux-arm64.tar.gz" was not found in "/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-e495ca/downloads/elastic-agent-9.4.0-SNAPSHOT-linux-arm64.tar.gz.sha512"
For help, please see our troubleshooting guide at https://www.elastic.co/docs/troubleshoot/ingest/fleet/common-problems

[michel-laterman]

Nope, the sha512 is already handled as part of the fs downloade:

elastic-agent/internal/pkg/agent/application/upgrade/artifact/download/fs/downloader.go

Lines 70 to 71 in 94595e0

	hashPath, err := e.download(e.config.OS(), a, *version, ".sha512")
	downloadedFiles = append(downloadedFiles, hashPath)

@ycombinator, the error you're seeing is indicating a (silent) permissions issue; the source-uri path does not match the path in the error message (it silently falls back to paths.Download if there is a permissions issue).

[ycombinator]

@ycombinator, the error you're seeing is indicating a (silent) permissions issue; the source-uri path does not match the path in the error message (it silently falls back to paths.Download if there is a permissions issue).

The path I specified in the --source-uri was file://$(pwd)/development/github/elastic-agent/build/distributions/. Here are the permissions for that path:

ls -al $(pwd)/development/github/elastic-agent/build/distributions/
total 410376
drwxr-xr-x 1 shaunak shaunak       128 Dec 19 02:09 .
drwxr-xr-x 1 shaunak shaunak       224 Dec 19 02:00 ..
-rw-r--r-- 1 shaunak shaunak 407341646 Dec 19 02:03 elastic-agent-9.4.0-SNAPSHOT-linux-arm64.tar.gz
-rw-r--r-- 1 shaunak shaunak       186 Dec 19 02:03 elastic-agent-9.4.0-SNAPSHOT-linux-arm64.tar.gz.sha512

As you can see, the .tar.gz.sha512 file is world-readable. And in any case, the elastic-agent command was executed with sudo. So I'm not following why this would be a permissions issue. 🤔

What changes should I make to file permissions to make the issue go away so that the following command would not error out on the .sha512 file (I realize it will error out due to the lack of a .asc file but that's a separate issue):

$ sudo elastic-agent upgrade --force --source-uri file://$(pwd)/development/github/elastic-agent/build/distributions/ 9.4.0-SNAPSHOT

[michel-laterman]

Using a multipass VM, i'm unable to recreate:

ubuntu@fleet-server-dev:~$ ls -la
total 407660
drwxr-x--- 9 ubuntu ubuntu      4096 Dec 19 12:37 .
drwxr-xr-x 3 root   root        4096 Dec 19 12:08 ..
-rw-rw-r-- 1 ubuntu ubuntu 417380783 Dec 15 22:07 elastic-agent-9.3.0-SNAPSHOT-linux-arm64.tar.gz
-rw-rw-r-- 1 ubuntu ubuntu       488 Dec 15 23:33 elastic-agent-9.3.0-SNAPSHOT-linux-arm64.tar.gz.asc
-rw-rw-r-- 1 ubuntu ubuntu       178 Dec 15 23:33 elastic-agent-9.3.0-SNAPSHOT-linux-arm64.tar.gz.sha512
ubuntu@fleet-server-dev:~$ sudo elastic-agent upgrade 9.3.0-SNAPSHOT --force --source-uri file://$(pwd)
Upgrade triggered to version 9.3.0-SNAPSHOT, Elastic Agent is currently restarting
ubuntu@fleet-server-dev:~$ sudo elastic-agent status
┌─ fleet
│  └─ status: (STARTING)
├─ elastic-agent
│  └─ status: (HEALTHY) Running
└─ upgrade_details
   ├─ target_version: 9.3.0
   ├─ state: UPG_WATCHING
   └─ metadata
ubuntu@fleet-server-dev:~$ sudo elastic-agent version
Binary: 9.3.0-SNAPSHOT (build: 612bfcd1e89ee79f41243a1b005cd33f4c42d4ab at 2025-12-16 01:47:25 +0000 UTC)
Daemon: 9.3.0-SNAPSHOT (build: 612bfcd1e89ee79f41243a1b005cd33f4c42d4ab at 2025-12-16 01:47:25 +0000 UTC)

I've even altered the permissions to what you have in order to try:

ubuntu@fleet-server-dev:~$ ls -la
total 407660
drwxr-x--- 9 ubuntu ubuntu      4096 Dec 19 12:37 .
drwxr-xr-x 3 root   root        4096 Dec 19 12:08 ..
-rw-r--r-- 1 ubuntu ubuntu 417380783 Dec 15 22:07 elastic-agent-9.3.0-SNAPSHOT-linux-arm64.tar.gz
-rw-rw-r-- 1 ubuntu ubuntu       488 Dec 15 23:33 elastic-agent-9.3.0-SNAPSHOT-linux-arm64.tar.gz.asc
-rw-r--r-- 1 ubuntu ubuntu       178 Dec 15 23:33 elastic-agent-9.3.0-SNAPSHOT-linux-arm64.tar.gz.sha512
ubuntu@fleet-server-dev:~$ sudo elastic-agent version
Binary: 9.3.0-SNAPSHOT (build: e495cacfae09aec99cf7c888faa797e9f55773c6 at 2025-12-19 18:14:33 +0000 UTC)
Daemon: 9.3.0-SNAPSHOT (build: e495cacfae09aec99cf7c888faa797e9f55773c6 at 2025-12-19 18:14:33 +0000 UTC)
ubuntu@fleet-server-dev:~$ sudo elastic-agent upgrade 9.3.0-SNAPSHOT --force --source-uri file://$(pwd)
Upgrade triggered to version 9.3.0-SNAPSHOT, Elastic Agent is currently restarting
ubuntu@fleet-server-dev:~$ sudo elastic-agent version
Binary: 9.3.0-SNAPSHOT (build: 612bfcd1e89ee79f41243a1b005cd33f4c42d4ab at 2025-12-16 01:47:25 +0000 UTC)
Daemon: 9.3.0-SNAPSHOT (build: 612bfcd1e89ee79f41243a1b005cd33f4c42d4ab at 2025-12-16 01:47:25 +0000 UTC)
ubuntu@fleet-server-dev:~$ sudo elastic-agent status
┌─ fleet
│  └─ status: (STARTING)
├─ elastic-agent
│  └─ status: (HEALTHY) Running
└─ upgrade_details
   ├─ target_version: 9.3.0
   ├─ state: UPG_WATCHING
   └─ metadata

But I'm not able to recreate your error at all.

Can you see what's in /opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-e495ca/downloads/ after it fails for you?

[ycombinator]

Can you see what's in /opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-e495ca/downloads/ after it fails for you?

$ whoami
root
$ ls -al /opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-e495ca/downloads/
total 4
drwxr-x--- 1 root root   0 Dec 21 19:53 .
drwxr-x--- 1 root root 166 Dec 21 19:53 ..

I turned on debug level logging and retried the upgrade command. I see these logs:

{"log.level":"info","@timestamp":"2025-12-22T03:56:18.144Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).logUpgradeDetails","file.name":"coordinator/coordinator.go","file.line":899},"message":"updated upgrade details","log":{"source":"elastic-agent"},"upgrade_details":{"target_version":"9.4.0-SNAPSHOT","state":"UPG_REQUESTED","metadata":{}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-12-22T03:56:18.145Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Upgrade","file.name":"upgrade/upgrade.go","file.line":289},"message":"Upgrading agent","log":{"source":"elastic-agent"},"version":"9.4.0-SNAPSHOT","source_uri":"file:///home/shaunak","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-12-22T03:56:18.145Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.cleanNonMatchingVersionsFromDownloads","file.name":"upgrade/cleanup.go","file.line":21},"message":"Cleaning up non-matching downloaded versions","log":{"source":"elastic-agent"},"version":"9.3.0","downloads.path":"/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-e495ca/downloads","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-12-22T03:56:18.145Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*artifactDownloader).appendFallbackPGP","file.name":"upgrade/step_download.go","file.line":159},"message":"Considering fleet server uri for pgp check fallback \"\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-12-22T03:56:18.145Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*artifactDownloader).downloadArtifact","file.name":"upgrade/step_download.go","file.line":103},"message":"Using local upgrade artifact","log":{"source":"elastic-agent"},"version":"9.4.0-SNAPSHOT","drop_path":"/home/shaunak","target_path":"/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-e495ca/downloads","install_path":"/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-e495ca/install","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-12-22T03:56:18.145Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).logUpgradeDetails","file.name":"coordinator/coordinator.go","file.line":899},"message":"updated upgrade details","log":{"source":"elastic-agent"},"upgrade_details":{"target_version":"9.4.0-SNAPSHOT","state":"UPG_DOWNLOADING","metadata":{}},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-12-22T03:56:18.145Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade/artifact/download.VerifySHA512HashWithCleanup","file.name":"download/verifier.go","file.line":114},"message":"error verifying the package using hash file \"/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-e495ca/downloads/elastic-agent-9.4.0-SNAPSHOT-linux-arm64.tar.gz.sha512\", contents: \"32f6feefaa4734f61e001b7ebb19735955a6b18e9cd6c3aa41ba0158706527cf417f1a2ba6f2842f4c274b9ec52f3cf2cab684792194caebd6f53a979373f6a6  elastic-agent-9.4.0-SNAPSHOT-SNAPSHOT-linux-arm64.tar.gz\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-12-22T03:56:18.145Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.cleanNonMatchingVersionsFromDownloads","file.name":"upgrade/cleanup.go","file.line":21},"message":"Cleaning up non-matching downloaded versions","log":{"source":"elastic-agent"},"version":"9.3.0","downloads.path":"/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-e495ca/downloads","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-12-22T03:56:18.145Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).Upgrade","file.name":"coordinator/coordinator.go","file.line":873},"message":"upgrade failed","log":{"source":"elastic-agent"},"error":{"Key":"error","Type":26,"Integer":0,"String":"","Interface":{}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-12-22T03:56:18.145Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).logUpgradeDetails","file.name":"coordinator/coordinator.go","file.line":899},"message":"updated upgrade details","log":{"source":"elastic-agent"},"upgrade_details":{"target_version":"9.4.0-SNAPSHOT","state":"UPG_FAILED","metadata":{"failed_state":"UPG_DOWNLOADING","error_msg":"failed verification of agent binary: failed to verify SHA512 hash: could not read checksum file: checksum for \"elastic-agent-9.4.0-SNAPSHOT-linux-arm64.tar.gz\" was not found in \"/opt/Elastic/Agent/data/elastic-agent-9.3.0-SNAPSHOT-e495ca/downloads/elastic-agent-9.4.0-SNAPSHOT-linux-arm64.tar.gz.sha512\""}},"ecs.version":"1.6.0"}

[ycombinator]

Interestingly, I don't run into any issues if I download the target version's .tar.gz, .sha512, and .asc files from https://www.elastic.co/downloads/elastic-agent. The problems I was running into only seem to happen if I've built the target version from source. 🤷

Anyway, I'm able to test the changes in this PR now, so I'm unblocked.

[github-actions]

@Mergifyio backport 8.19 9.1 9.2 9.3

[mergify]

backport 8.19 9.1 9.2 9.3

✅ Backports have been created

Details

• #11975 [8.19] (backport #11826) Fix signature verification using the upgrade command with the --source-uri flag for fleet-managed agents has been created for branch 8.19
• #11976 [9.1] (backport #11826) Fix signature verification using the upgrade command with the --source-uri flag for fleet-managed agents has been created for branch 9.1
• #11977 [9.2] (backport #11826) Fix signature verification using the upgrade command with the --source-uri flag for fleet-managed agents has been created for branch 9.2
• #11978 [9.3] (backport #11826) Fix signature verification using the upgrade command with the --source-uri flag for fleet-managed agents has been created for branch 9.3

[nimarezainia]

@tlee-elastic fyi