Fix Filebeat's race condition on shutdown

[belimawr]

Note for reviewers

The detailed description of the issue this PR solves can be found at #45034 (comment)

Proposed commit message

When Filebeat fails to start because of a unknown input, it can get into a deadlock state and never exit. This commit attempts to fix it.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

## Disruptive User Impact
## Author's Checklist

How to test this PR locally

That's a non-deterministic problem, so testing is about running Filebeat over and over again and ensuring it never hangs during shutdown.

You can file the reproducing steps listed in the linked issue below, to run Filebeat over and over again, use the following script:

#!/bin/bash

# while "$@"; do :; done
counter=0
while true; do
    "$@"
    let counter++
    echo $counter
    true
done

Assuming the script is named rununtilfail and is ~/bin/rununtilfail , you can run:

~/bin/rununtilfail ./filebeat -c ./filebeat-test.yml -e

Let it run for a few minutes, if Filebeat never hangs, then it's very likely the issue has been solved.

On my tests, it usually took less than 100 runs for Filebeat to hang, in more extreme cases it would take a couple thousand tries for Filebeat to hang.

Related issues

• Closes Filebeat sometimes does not shut down when only one input is configured and it is non-existent #45034

## Use cases
## Screenshots
## Logs

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[mauri870]

Reminder for me to check if this somehow fixes #43137.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[github-actions]

@Mergifyio backport 8.17 8.18 8.19 9.0 9.1

[mergify]

backport 8.17 8.18 8.19 9.0 9.1

✅ Backports have been created

Details

• #46393 [8.17](backport #46331) Fix Filebeat's race condition on shutdown has been created for branch 8.17
• #46394 [8.18](backport #46331) Fix Filebeat's race condition on shutdown has been created for branch 8.18
• #46395 [8.19](backport #46331) Fix Filebeat's race condition on shutdown has been created for branch 8.19
• #46396 [9.0](backport #46331) Fix Filebeat's race condition on shutdown has been created for branch 9.0
• #46397 [9.1](backport #46331) Fix Filebeat's race condition on shutdown has been created for branch 9.1