Skip to content

fix: package fleet-server using golang-crossbuild #5392

Merged
pkoutsovasilis merged 4 commits intoelastic:mainfrom
pkoutsovasilis:fix/glibc_dependencies
Sep 5, 2025
Merged

fix: package fleet-server using golang-crossbuild #5392
pkoutsovasilis merged 4 commits intoelastic:mainfrom
pkoutsovasilis:fix/glibc_dependencies

Conversation

@pkoutsovasilis
Copy link
Contributor

@pkoutsovasilis pkoutsovasilis commented Sep 3, 2025 •
edited
Loading

What is the problem this PR solves?

The fleet-server binary was previously built directly on VM runners instead of using a controlled environment such as golang-crossbuild.
This caused the binary to be dynamically linked against the host system’s GLIBC, introducing higher GLIBC requirements (≥ 2.32) than those declared in the Elastic OS support matrix (≥ 2.17).

Note that, even if the build configuration used CGO_ENABLED=0 (which didn't before this PR) together with -buildmode=pie, on Linux, this combination still produces a dynamically linked binary because pie relies on the dynamic loader (e.g. /lib/ld-linux-aarch64.so.1 on arm64).

How does this PR solve the problem?

  • Switches builds for all packaging steps to run inside golang-crossbuild images (via mage docker:release), avoiding linkage against host GLIBC.
  • Forces CGO_ENABLED=0 across builds (except for FIPS variants, which require CGO).
  • Removes -buildmode=pie so that binaries are fully static, with no interpreter requirement.
  • Updates Buildkite pipelines to:
    • Add explicit packaging steps for x86_64 and arm64 (including FIPS variants).
    • Save packaging artifacts (build/distributions/**) for DRA validation and debugging.
  • Fixes an issue with mage docker:image when compiling FIPS variants.

As a result, Fleet Server binaries are now fully static and do not have a GLIBC requirement, restoring compatibility with Amazon Linux 2 and other supported distributions.

For example, you can download the Linux packaging artifacts from the CI steps introduced in this PR:

Running file against them after extraction shows static binaries:

$ file ~/Downloads/fleet-server-9.2.0-linux-arm64/fleet-server
/Users/pkoutsovasilis/Downloads/fleet-server-9.2.0-linux-arm64/fleet-server: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=22095f2a8f3da7f6271dd858dcc13fe86e19acf2, stripped

$ file ~/Downloads/fleet-server-9.2.0-linux-x86_64/fleet-server
/Users/pkoutsovasilis/Downloads/fleet-server-9.2.0-linux-x86_64/fleet-server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=f9a203f5c78161f88d9e10be1bf5863f970adcac, stripped

How to test this PR locally

mage docker:binary
docker run --rm -it --entrypoint /app/fleet-server -v $PWD/build/binaries/fleet-server-9.2.0-linux-arm64:/app amazonlinux:2 --version

Design Checklist

  • I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
  • I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
  • I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool

Related issues
@pkoutsovasilis pkoutsovasilis self-assigned this Sep 3, 2025
@pkoutsovasilis pkoutsovasilis added bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team backport-active-all Automated backport with mergify to all the active branches skip-changelog labels Sep 3, 2025
@prodsecmachine
Copy link

prodsecmachine commented Sep 3, 2025 •
edited
Loading

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)
@pkoutsovasilis pkoutsovasilis force-pushed the fix/glibc_dependencies branch 4 times, most recently from 4d7b666 to 08e28d8 Compare September 3, 2025 19:13
@pkoutsovasilis pkoutsovasilis marked this pull request as ready for review September 3, 2025 23:01
@pkoutsovasilis pkoutsovasilis requested a review from a team as a code owner September 3, 2025 23:01
@elastic-sonarqube
Copy link

elastic-sonarqube bot commented Sep 3, 2025

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube
@pkoutsovasilis pkoutsovasilis merged commit 7da3f69 into elastic:main Sep 5, 2025
9 checks passed
@github-actions
Copy link
Contributor

github-actions bot commented Sep 5, 2025

@Mergifyio backport 8.18 8.19 9.0 9.1
@mergify
Copy link
Contributor

mergify bot commented Sep 5, 2025 •
edited
Loading

backport 8.18 8.19 9.0 9.1

✅ Backports have been created

Details
mergify bot pushed a commit that referenced this pull request Sep 5, 2025
@pkoutsovasilis @mergify
fix: package fleet-server using golang-crossbuild (#5392)
26f6d40 
* fix: package fleet-server using golang-crossbuild and introduce packaging steps on every PR

* feat: save artifacts during packaging for DRA

* fix: disable CGO and buildmode pie

* doc: add changelog fragment

(cherry picked from commit 7da3f69)
mergify bot pushed a commit that referenced this pull request Sep 5, 2025
@pkoutsovasilis @mergify
fix: package fleet-server using golang-crossbuild (#5392)
f11466c 
* fix: package fleet-server using golang-crossbuild and introduce packaging steps on every PR

* feat: save artifacts during packaging for DRA

* fix: disable CGO and buildmode pie

* doc: add changelog fragment

(cherry picked from commit 7da3f69)

# Conflicts:
#	.buildkite/pipeline.package.mbp.yml
#	.buildkite/pipeline.yml
#	.buildkite/scripts/package.sh
#	Dockerfile.fips
#	magefile.go
@mergify mergify bot mentioned this pull request Sep 5, 2025
Merged
8 tasks
mergify bot pushed a commit that referenced this pull request Sep 5, 2025
@pkoutsovasilis @mergify
fix: package fleet-server using golang-crossbuild (#5392)
7742c90 
* fix: package fleet-server using golang-crossbuild and introduce packaging steps on every PR

* feat: save artifacts during packaging for DRA

* fix: disable CGO and buildmode pie

* doc: add changelog fragment

(cherry picked from commit 7da3f69)

# Conflicts:
#	.buildkite/pipeline.package.mbp.yml
#	.buildkite/pipeline.yml
#	.buildkite/scripts/package.sh
#	Dockerfile.fips
#	magefile.go
This was referenced Sep 5, 2025
Merged
Merged
mergify bot pushed a commit that referenced this pull request Sep 5, 2025
@pkoutsovasilis @mergify
fix: package fleet-server using golang-crossbuild (#5392)
04323f8 
* fix: package fleet-server using golang-crossbuild and introduce packaging steps on every PR

* feat: save artifacts during packaging for DRA

* fix: disable CGO and buildmode pie

* doc: add changelog fragment

(cherry picked from commit 7da3f69)
@mergify mergify bot mentioned this pull request Sep 5, 2025
Merged
8 tasks
pkoutsovasilis added a commit that referenced this pull request Sep 5, 2025
@mergify @pkoutsovasilis
fix: package fleet-server using golang-crossbuild (#5392) (#5430)
701081b 
* fix: package fleet-server using golang-crossbuild and introduce packaging steps on every PR

* feat: save artifacts during packaging for DRA

* fix: disable CGO and buildmode pie

* doc: add changelog fragment

(cherry picked from commit 7da3f69)

Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>
pkoutsovasilis added a commit that referenced this pull request Sep 5, 2025
@mergify @pkoutsovasilis
fix: package fleet-server using golang-crossbuild (#5392) (#5428)
158009a 
* fix: package fleet-server using golang-crossbuild and introduce packaging steps on every PR

* feat: save artifacts during packaging for DRA

* fix: disable CGO and buildmode pie

* doc: add changelog fragment

(cherry picked from commit 7da3f69)

Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>
@pkoutsovasilis pkoutsovasilis mentioned this pull request Sep 5, 2025
Merged
8 tasks
This was referenced Sep 5, 2025
Closed
Merged
Closed
Merged
pkoutsovasilis added a commit that referenced this pull request Sep 8, 2025
@mergify @pkoutsovasilis
[9.0](backport #5392) fix: package fleet-server using golang-crossbui…
5054259 
…ld (#5429)

* fix: package fleet-server using golang-crossbuild  (#5392)

* fix: package fleet-server using golang-crossbuild and introduce packaging steps on every PR

* feat: save artifacts during packaging for DRA

* fix: disable CGO and buildmode pie

* doc: add changelog fragment

(cherry picked from commit 7da3f69)

# Conflicts:
#	.buildkite/pipeline.package.mbp.yml
#	.buildkite/pipeline.yml
#	.buildkite/scripts/package.sh
#	Dockerfile.fips
#	magefile.go

* fix: adjust changes for make

* fix: fleet-server-package-mbp CI step dependencies (#5446)

(cherry picked from commit 5aa10ef)

---------

Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>
pkoutsovasilis added a commit that referenced this pull request Sep 8, 2025
@mergify @pkoutsovasilis
[8.18](backport #5392) fix: package fleet-server using golang-crossbu…
1a5ec7f 
…ild (#5427)

* fix: package fleet-server using golang-crossbuild  (#5392)

* fix: package fleet-server using golang-crossbuild and introduce packaging steps on every PR

* feat: save artifacts during packaging for DRA

* fix: disable CGO and buildmode pie

* doc: add changelog fragment

(cherry picked from commit 7da3f69)

# Conflicts:
#	.buildkite/pipeline.package.mbp.yml
#	.buildkite/pipeline.yml
#	.buildkite/scripts/package.sh
#	Dockerfile.fips
#	magefile.go

* fix: adjust changes for make

* fix: fleet-server-package-mbp CI step dependencies (#5446)

(cherry picked from commit 5aa10ef)

---------

Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>
@v1v v1v mentioned this pull request Nov 14, 2025
Merged
8 tasks
This was referenced Nov 14, 2025
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-active-all Automated backport with mergify to all the active branches bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

4 participants

@pkoutsovasilis @prodsecmachine @ycombinator @ebeahan