Remove ID from list when Filestream inputs fail to be created by belimawr · Pull Request #44697 · elastic/beats

belimawr · 2025-06-06T21:29:47Z

Proposed commit message

Filestream was not removing the ID from inputs that failed to start, that made re-starting the same input with a valid configuration to fail with a duplicated ID error. This affects any configuration mode that can be reloaded, Elastic-Agent being the main use case.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

~~## Author's Checklist~~

How to test this PR locally

1. Create the following files

filebeat.yml

filebeat.config:
  inputs:
    enabled: true
    path: inputs.d/*.yml
    reload.enabled: true
    reload.period: 2s

queue.mem:
  flush:
    timeout: 0s

output.console:
  pretty: true

inputs.d/filestream.yml

- type: filestream
  id: t-wing
  paths:
    - /**/**/log.log
    # - /tmp/flog.log

/tmp/flog.log

<90>2 2025-06-09T12:31:10.044Z internalworld-class.net sed 3808 ID961 - We need to connect the primary THX protocol!
<62>2 2025-06-09T12:31:10.044Z investorfunctionalities.name velit 2310 ID769 - The SSL monitor is down, hack the open-source capacitor so we can quantify the PCI system!
<49>1 2025-06-09T12:31:10.044Z nationalmatrix.net rerum 798 ID992 - We need to compress the back-end TCP interface!
<187>1 2025-06-09T12:31:10.044Z humanmesh.org dolores 2059 ID720 - Use the multi-byte TCP protocol, then you can program the wireless array!
<188>2 2025-06-09T12:31:10.044Z districtsolutions.net in 333 ID581 - Use the haptic PNG pixel, then you can input the redundant transmitter!
<17>1 2025-06-09T12:31:10.044Z legacycross-media.net sunt 1957 ID621 - Try to quantify the SDD system, maybe it will quantify the solid state capacitor!
<54>1 2025-06-09T12:31:10.044Z internalmesh.io et 9739 ID142 - Hacking the system won't do anything, we need to parse the primary RAM hard drive!
<161>2 2025-06-09T12:31:10.044Z regionalsexy.net assumenda 9612 ID649 - You can't bypass the firewall without bypassing the back-end AI sensor!
<131>2 2025-06-09T12:31:10.044Z futureembrace.info labore 900 ID122 - I'Ll generate the virtual XSS circuit, that should array the USB sensor!
<6>2 2025-06-09T12:31:10.044Z principalrepurpose.net provident 8513 ID99 - Try to hack the SMS interface, maybe it will back up the bluetooth program!

2. Build and Start Filebeat

go build .
./filebeat

Wait for the log error:

Error creating runner from config: cannot create prospector: error while creating filewatcher multiple ** in \"/**/**/log.log\"

3. Edit `inputs.d/filestream.yml` so it is valid

Remove the first, invalid, path and uncomment the second one. paths section so it looks like this:

  paths:
    #- /**/**/log.log
    - /tmp/flog.log

Save the file and wait to see the ingested events in the console.

~~## Related issues~~
~~## Use cases~~
~~## Screenshots~~
~~## Logs~~

Filestream was not removing the ID from inputs that failed to start, that made re-starting the same input with a valid configuration to fail with a duplicated ID error. This affects any configuration mode that can be reloaded, Elastic-Agent being the main use case.

github-actions · 2025-06-06T21:29:58Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2025-06-06T21:30:26Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

filebeat/input/filestream/internal/input-logfile/manager.go

elasticmachine · 2025-06-09T16:45:37Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

filebeat/input/filestream/internal/input-logfile/manager.go

While duplicated IDs should never happen, we need to allow it for legacy reasons, so when handling a failed input we need to consider that there might be another input with the same ID running. When we check if the ID is already in use we set flag, `idAlreadyInUse` and we only remove the ID in a failure scenario if the flag is false.

filebeat/input/filestream/internal/input-logfile/manager_test.go

Co-authored-by: Anderson Queiroz <me@andersonq.me>

belimawr · 2025-06-10T12:13:41Z

I've enabled auto-merge.

Thanks for the quick review folks!

github-actions · 2025-06-10T18:47:46Z

@Mergifyio backport 8.17 8.18 8.19 9.0

mergify · 2025-06-10T18:48:13Z

backport 8.17 8.18 8.19 9.0

✅ Backports have been created

Details

#44739 [8.17](backport #44697) Remove ID from list when Filestream inputs fail to be created has been created for branch 8.17 but encountered conflicts
#44737 [8.18](backport #44697) Remove ID from list when Filestream inputs fail to be created has been created for branch 8.18 but encountered conflicts
#44738 [8.19](backport #44697) Remove ID from list when Filestream inputs fail to be created has been created for branch 8.19 but encountered conflicts
#44740 [9.0](backport #44697) Remove ID from list when Filestream inputs fail to be created has been created for branch 9.0 but encountered conflicts

Filestream was not removing the ID from inputs that failed to start, that made re-starting the same input with a valid configuration to fail with a duplicated ID error. This affects any configuration mode that can be reloaded, Elastic-Agent being the main use case. This commit fixes it by removing the ID from a failed input. --------- Co-authored-by: Anderson Queiroz <me@andersonq.me> (cherry picked from commit 017cc4f) # Conflicts: # filebeat/input/filestream/internal/input-logfile/manager.go # filebeat/input/filestream/internal/input-logfile/manager_test.go

Filestream was not removing the ID from inputs that failed to start, that made re-starting the same input with a valid configuration to fail with a duplicated ID error. This affects any configuration mode that can be reloaded, Elastic-Agent being the main use case. This commit fixes it by removing the ID from a failed input. --------- Co-authored-by: Anderson Queiroz <me@andersonq.me> (cherry picked from commit 017cc4f) # Conflicts: # filebeat/input/filestream/internal/input-logfile/manager.go

…l to be created (#44740) Filestream was not removing the ID from inputs that failed to start, that made re-starting the same input with a valid configuration to fail with a duplicated ID error. This affects any configuration mode that can be reloaded, Elastic-Agent being the main use case. This commit fixes it by removing the ID from a failed input. --------- Co-authored-by: Anderson Queiroz <me@andersonq.me> (cherry picked from commit 017cc4f) # Conflicts: # filebeat/input/filestream/internal/input-logfile/manager.go * fix merge conflicts --------- Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

…il to be created (#44738) Filestream was not removing the ID from inputs that failed to start, that made re-starting the same input with a valid configuration to fail with a duplicated ID error. This affects any configuration mode that can be reloaded, Elastic-Agent being the main use case. This commit fixes it by removing the ID from a failed input. --------- Co-authored-by: Anderson Queiroz <me@andersonq.me> (cherry picked from commit 017cc4f) # Conflicts: # filebeat/input/filestream/internal/input-logfile/manager.go # filebeat/input/filestream/internal/input-logfile/manager_test.go * Fix merge conflicts --------- Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

…il to be created (#44737) Filestream was not removing the ID from inputs that failed to start, that made re-starting the same input with a valid configuration to fail with a duplicated ID error. This affects any configuration mode that can be reloaded, Elastic-Agent being the main use case. This commit fixes it by removing the ID from a failed input. --------- Co-authored-by: Anderson Queiroz <me@andersonq.me> (cherry picked from commit 017cc4f) # Conflicts: # filebeat/input/filestream/internal/input-logfile/manager.go # filebeat/input/filestream/internal/input-logfile/manager_test.go * Fix merge conflicts --------- Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

…il to be created (#44739) Filestream was not removing the ID from inputs that failed to start, that made re-starting the same input with a valid configuration to fail with a duplicated ID error. This affects any configuration mode that can be reloaded, Elastic-Agent being the main use case. This commit fixes it by removing the ID from a failed input. --------- Co-authored-by: Anderson Queiroz <me@andersonq.me> (cherry picked from commit 017cc4f) # Conflicts: # filebeat/input/filestream/internal/input-logfile/manager.go # filebeat/input/filestream/internal/input-logfile/manager_test.go * Fix merge conflicts --------- Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 6, 2025

mergify bot assigned belimawr Jun 6, 2025

belimawr commented Jun 6, 2025

View reviewed changes

filebeat/input/filestream/internal/input-logfile/manager.go Show resolved Hide resolved

belimawr added 2 commits June 9, 2025 12:22

Add a test

17551af

Add changelog entry

00e19e1

belimawr added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Jun 9, 2025

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 9, 2025

belimawr added needs_team Indicates that the issue/PR needs a Team:* label backport-active-8 Automated backport with mergify to all the active 8.[0-9]+ branches backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches labels Jun 9, 2025

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 9, 2025

belimawr added needs_team Indicates that the issue/PR needs a Team:* label bugfix labels Jun 9, 2025

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 9, 2025

belimawr marked this pull request as ready for review June 9, 2025 16:45

belimawr requested a review from a team as a code owner June 9, 2025 16:45

belimawr requested review from AndersonQ and mauri870 June 9, 2025 16:45

leehinman reviewed Jun 9, 2025

View reviewed changes

filebeat/input/filestream/internal/input-logfile/manager.go Outdated Show resolved Hide resolved

belimawr added 2 commits June 9, 2025 14:45

Add tests for failure paths

16978af

belimawr requested a review from leehinman June 9, 2025 19:14

leehinman approved these changes Jun 9, 2025

View reviewed changes

AndersonQ approved these changes Jun 10, 2025

View reviewed changes

filebeat/input/filestream/internal/input-logfile/manager_test.go Outdated Show resolved Hide resolved

Update filebeat/input/filestream/internal/input-logfile/manager_test.go

28d66b5

Co-authored-by: Anderson Queiroz <me@andersonq.me>

belimawr enabled auto-merge (squash) June 10, 2025 12:13

mauri870 approved these changes Jun 10, 2025

View reviewed changes

belimawr disabled auto-merge June 10, 2025 18:45

belimawr merged commit 017cc4f into elastic:main Jun 10, 2025
46 of 50 checks passed

belimawr deleted the fix-filestream-creation-issue branch June 10, 2025 18:47

mergify bot mentioned this pull request Jun 10, 2025

[8.18](backport #44697) Remove ID from list when Filestream inputs fail to be created #44737

Merged

4 tasks

This was referenced Jun 10, 2025

[8.19](backport #44697) Remove ID from list when Filestream inputs fail to be created #44738

Merged

[8.17](backport #44697) Remove ID from list when Filestream inputs fail to be created #44739

Merged

mergify bot mentioned this pull request Jun 10, 2025

[9.0](backport #44697) Remove ID from list when Filestream inputs fail to be created #44740

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove ID from list when Filestream inputs fail to be created#44697

Remove ID from list when Filestream inputs fail to be created#44697
belimawr merged 6 commits intoelastic:mainfrom
belimawr:fix-filestream-creation-issue

belimawr commented Jun 6, 2025 •

edited

Loading

github-actions bot commented Jun 6, 2025

mergify bot commented Jun 6, 2025

Uh oh!

elasticmachine commented Jun 9, 2025

Uh oh!

Uh oh!

belimawr commented Jun 10, 2025

Uh oh!

github-actions bot commented Jun 10, 2025

mergify bot commented Jun 10, 2025 •

edited

Loading

Labels

5 participants

Conversation

belimawr commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Disruptive User Impact

How to test this PR locally

1. Create the following files

2. Build and Start Filebeat

3. Edit inputs.d/filestream.yml so it is valid

github-actions bot commented Jun 6, 2025

🤖 GitHub comments

mergify bot commented Jun 6, 2025

Uh oh!

elasticmachine commented Jun 9, 2025

Uh oh!

Uh oh!

belimawr commented Jun 10, 2025

Uh oh!

github-actions bot commented Jun 10, 2025

mergify bot commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Labels

5 participants

belimawr commented Jun 6, 2025 •

edited

Loading

3. Edit `inputs.d/filestream.yml` so it is valid

mergify bot commented Jun 10, 2025 •

edited

Loading