Skip to content

[8.18](backport #5392) fix: package fleet-server using golang-crossbuild #5427

Merged
pkoutsovasilis merged 3 commits into8.18from
mergify/bp/8.18/pr-5392
Sep 8, 2025
Merged

[8.18](backport #5392) fix: package fleet-server using golang-crossbuild #5427
pkoutsovasilis merged 3 commits into8.18from
mergify/bp/8.18/pr-5392

Conversation

@mergify
Copy link
Contributor

@mergify mergify bot commented Sep 5, 2025

What is the problem this PR solves?

The fleet-server binary was previously built directly on VM runners instead of using a controlled environment such as golang-crossbuild.
This caused the binary to be dynamically linked against the host system’s GLIBC, introducing higher GLIBC requirements (≥ 2.32) than those declared in the Elastic OS support matrix (≥ 2.17).

Note that, even if the build configuration used CGO_ENABLED=0 (which didn't before this PR) together with -buildmode=pie, on Linux, this combination still produces a dynamically linked binary because pie relies on the dynamic loader (e.g. /lib/ld-linux-aarch64.so.1 on arm64).

How does this PR solve the problem?

  • Switches builds for all packaging steps to run inside golang-crossbuild images (via mage docker:release), avoiding linkage against host GLIBC.
  • Forces CGO_ENABLED=0 across builds (except for FIPS variants, which require CGO).
  • Removes -buildmode=pie so that binaries are fully static, with no interpreter requirement.
  • Updates Buildkite pipelines to:
    • Add explicit packaging steps for x86_64 and arm64 (including FIPS variants).
    • Save packaging artifacts (build/distributions/**) for DRA validation and debugging.
  • Fixes an issue with mage docker:image when compiling FIPS variants.

As a result, Fleet Server binaries are now fully static and do not have a GLIBC requirement, restoring compatibility with Amazon Linux 2 and other supported distributions.

For example, you can download the Linux packaging artifacts from the CI steps introduced in this PR:

Running file against them after extraction shows static binaries:

$ file ~/Downloads/fleet-server-9.2.0-linux-arm64/fleet-server
/Users/pkoutsovasilis/Downloads/fleet-server-9.2.0-linux-arm64/fleet-server: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=22095f2a8f3da7f6271dd858dcc13fe86e19acf2, stripped

$ file ~/Downloads/fleet-server-9.2.0-linux-x86_64/fleet-server
/Users/pkoutsovasilis/Downloads/fleet-server-9.2.0-linux-x86_64/fleet-server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=f9a203f5c78161f88d9e10be1bf5863f970adcac, stripped

How to test this PR locally

mage docker:binary
docker run --rm -it --entrypoint /app/fleet-server -v $PWD/build/binaries/fleet-server-9.2.0-linux-arm64:/app amazonlinux:2 --version

Design Checklist

  • I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
  • I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
  • I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool

Related issues


This is an automatic backport of pull request #5392 done by [Mergify](https://mergify.com).
* fix: package fleet-server using golang-crossbuild and introduce packaging steps on every PR

* feat: save artifacts during packaging for DRA

* fix: disable CGO and buildmode pie

* doc: add changelog fragment

(cherry picked from commit 7da3f69)

# Conflicts:
#	.buildkite/pipeline.package.mbp.yml
#	.buildkite/pipeline.yml
#	.buildkite/scripts/package.sh
#	Dockerfile.fips
#	magefile.go
@mergify mergify bot requested a review from a team as a code owner September 5, 2025 08:25
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Sep 5, 2025
@mergify mergify bot requested review from michel-laterman and pchila September 5, 2025 08:25
@mergify
Copy link
Contributor Author

mergify bot commented Sep 5, 2025

Cherry-pick of 7da3f69 has failed:

On branch mergify/bp/8.18/pr-5392
Your branch is up to date with 'origin/8.18'.

You are currently cherry-picking commit 7da3f69.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1756940128-fix-binary-compilation.yaml

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   .buildkite/pipeline.package.mbp.yml
	both modified:   .buildkite/pipeline.yml
	both modified:   .buildkite/scripts/package.sh
	both modified:   Dockerfile.fips
	deleted by us:   magefile.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@prodsecmachine
Copy link

prodsecmachine commented Sep 5, 2025

🎉 Snyk checks have passed. No issues have been found so far.

security/snyk check is complete. No issues have been found. (View Details)

license/snyk check is complete. No issues have been found. (View Details)

@github-actions github-actions bot added bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Sep 5, 2025
@mergify
Copy link
Contributor Author

mergify bot commented Sep 8, 2025

This pull request has not been merged yet. Could you please review and merge it @pkoutsovasilis? 🙏

@elastic-sonarqube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@pkoutsovasilis pkoutsovasilis merged commit 1a5ec7f into 8.18 Sep 8, 2025
8 checks passed
@pkoutsovasilis pkoutsovasilis deleted the mergify/bp/8.18/pr-5392 branch September 8, 2025 10:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport bug Something isn't working conflicts There is a conflict in the backported pull request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

2 participants