fix: blocking issues of runtime communicator by pkoutsovasilis · Pull Request #8881 · elastic/elastic-agent

pkoutsovasilis · 2025-07-08T06:45:37Z

What does this PR do?

This PR fixes multiple deadlock conditions in the runtime communicator logic and improves overall lifecycle handling. Specifically, it:

Handles potential deadlock when sending the initial observed message to the runtime if the communicator is destroyed.
Handles deadlock when the runtime calls CheckinExpected (init expected check-in) and the communicator is already destroyed.
Handles deadlock when the runtime calls CheckinExpected (init expected check-in) and the server has been disconnected.
Although highly unlikely due to existing synchronisation primitives, it doesn't block against multiple init checkin messages that arrive after the first that completed the init checkin process.
Removes a redundant goroutine in the checkin method, which also allows returning accurate gRPC status codes back to the client. (PS: this optimisation can be also applied for the actions handling)
Adds comprehensive unit tests covering all the above scenarios to prevent regressions.

Why is it important?

These fixes ensure the runtime communicator behaves predictably and safely across lifecycle boundaries like shutdown, reconnection, and concurrent access. Without these changes, users could experience hangs or lost check-in signals in rare but critical failure modes.

You can see that the previous implementation fails to complete these scenarios under test in CI here

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Disruptive User Impact

This is a bug-fix for internal lifecycle logic and is not expected to introduce changes in behaviour or configuration requirements for end users.

How to test this PR locally

mage unitTest

Related issues

Relates Elastic Agent not serving policy to Endpoint #7944

elasticmachine · 2025-07-08T10:54:50Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

blakerouse

Looks really good. I like all the additional testing. That will hopefully keep this from dead locks in the future as well.

I do have one comment that is inline. See if it make sense.

pkg/component/runtime/runtime_comm.go

elastic-sonarqube · 2025-07-09T09:07:23Z

Quality Gate passed

Issues
2 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
95.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-07-09T09:45:57Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: b211ef1

Failed CI Steps

x86_64:sudo: standalone-upgrade

History

💚 Build #23731 succeeded edc74ff
💔 Build #23728 failed 7fdbca2

cc @pkoutsovasilis

blakerouse

Thanks. With the latest change, this looks good.

github-actions · 2025-07-11T03:11:25Z

@Mergifyio backport 8.17 8.18 8.19 9.0 9.1

mergify · 2025-07-11T03:11:53Z

backport 8.17 8.18 8.19 9.0 9.1

✅ Backports have been created

Details

#8944 [8.17] (backport #8881) fix: blocking issues of runtime communicator has been created for branch 8.17
#8945 [8.18] (backport #8881) fix: blocking issues of runtime communicator has been created for branch 8.18
#8946 [8.19] (backport #8881) fix: blocking issues of runtime communicator has been created for branch 8.19
#8947 [9.0] (backport #8881) fix: blocking issues of runtime communicator has been created for branch 9.0
#8948 [9.1] (backport #8881) fix: blocking issues of runtime communicator has been created for branch 9.1

* ci: write unit-tests for runtime_comm.go * fix: blocking issues in runtime_comm.go * fix: QF1004 use strings.ReplaceAll * fix: guard closing c.runtimeCheckinDone with a local variable (cherry picked from commit 0d316ce)

* ci: write unit-tests for runtime_comm.go * fix: blocking issues in runtime_comm.go * fix: QF1004 use strings.ReplaceAll * fix: guard closing c.runtimeCheckinDone with a local variable (cherry picked from commit 0d316ce) Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>

…8945) * fix: blocking issues of runtime communicator (#8881) * ci: write unit-tests for runtime_comm.go * fix: blocking issues in runtime_comm.go * fix: QF1004 use strings.ReplaceAll * fix: guard closing c.runtimeCheckinDone with a local variable (cherry picked from commit 0d316ce) * fix: G115 linter error --------- Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>

* ci: write unit-tests for runtime_comm.go * fix: blocking issues in runtime_comm.go * fix: QF1004 use strings.ReplaceAll * fix: guard closing c.runtimeCheckinDone with a local variable (cherry picked from commit 0d316ce) Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>

…8944) * fix: blocking issues of runtime communicator (#8881) * ci: write unit-tests for runtime_comm.go * fix: blocking issues in runtime_comm.go * fix: QF1004 use strings.ReplaceAll * fix: guard closing c.runtimeCheckinDone with a local variable (cherry picked from commit 0d316ce) * fix: G115 linter error --------- Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>

* ci: write unit-tests for runtime_comm.go * fix: blocking issues in runtime_comm.go * fix: QF1004 use strings.ReplaceAll * fix: guard closing c.runtimeCheckinDone with a local variable (cherry picked from commit 0d316ce) Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>

ci: write unit-tests for runtime_comm.go

7fdbca2

pkoutsovasilis self-assigned this Jul 8, 2025

pkoutsovasilis added bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team backport-active-all Automated backport with mergify to all the active branches labels Jul 8, 2025

pkoutsovasilis added 2 commits July 8, 2025 10:50

fix: blocking issues in runtime_comm.go

254b401

fix: QF1004 use strings.ReplaceAll

edc74ff

pkoutsovasilis requested a review from blakerouse July 8, 2025 10:54

pkoutsovasilis marked this pull request as ready for review July 8, 2025 10:54

pkoutsovasilis requested a review from a team as a code owner July 8, 2025 10:54

pkoutsovasilis requested a review from michel-laterman July 8, 2025 10:54

pkoutsovasilis requested a review from cmacknz July 8, 2025 10:54

blakerouse reviewed Jul 8, 2025

View reviewed changes

pkg/component/runtime/runtime_comm.go Outdated Show resolved Hide resolved

fix: guard closing c.runtimeCheckinDone with a local variable

b211ef1

pkoutsovasilis requested a review from blakerouse July 9, 2025 12:50

blakerouse approved these changes Jul 10, 2025

View reviewed changes

pkoutsovasilis merged commit 0d316ce into elastic:main Jul 11, 2025
19 checks passed

pkoutsovasilis deleted the fix/runtime_comm_init_checkin branch July 11, 2025 03:11

mergify bot mentioned this pull request Jul 11, 2025

[8.17] (backport #8881) fix: blocking issues of runtime communicator #8944

Merged

8 tasks

mergify bot mentioned this pull request Jul 11, 2025

[8.18] (backport #8881) fix: blocking issues of runtime communicator #8945

Merged

8 tasks

mergify bot mentioned this pull request Jul 11, 2025

[8.19] (backport #8881) fix: blocking issues of runtime communicator #8946

Merged

8 tasks

mergify bot mentioned this pull request Jul 11, 2025

[9.0] (backport #8881) fix: blocking issues of runtime communicator #8947

Merged

8 tasks

mergify bot mentioned this pull request Jul 11, 2025

[9.1] (backport #8881) fix: blocking issues of runtime communicator #8948

Merged

8 tasks

pkoutsovasilis mentioned this pull request Jul 16, 2025

Elastic Agent not serving policy to Endpoint #7944

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: blocking issues of runtime communicator#8881

fix: blocking issues of runtime communicator#8881
pkoutsovasilis merged 4 commits intoelastic:mainfrom
pkoutsovasilis:fix/runtime_comm_init_checkin

pkoutsovasilis commented Jul 8, 2025 •

edited

Loading

elasticmachine commented Jul 8, 2025

blakerouse left a comment

Uh oh!

elastic-sonarqube bot commented Jul 9, 2025

elasticmachine commented Jul 9, 2025

blakerouse left a comment

Uh oh!

github-actions bot commented Jul 11, 2025

mergify bot commented Jul 11, 2025 •

edited

Loading

Labels

3 participants

Conversation

pkoutsovasilis commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

elasticmachine commented Jul 8, 2025

blakerouse left a comment

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube bot commented Jul 9, 2025

Quality Gate passed

elasticmachine commented Jul 9, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

History

blakerouse left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 11, 2025

mergify bot commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Labels

3 participants

pkoutsovasilis commented Jul 8, 2025 •

edited

Loading

mergify bot commented Jul 11, 2025 •

edited

Loading