Skip to content

x-pack/filebeat/input/streaming: fix crowdstrike cursor handling#44548

Merged
efd6 merged 3 commits intoelastic:mainfrom
efd6:44364-streaming_crowdstrike
Jun 12, 2025
Merged

x-pack/filebeat/input/streaming: fix crowdstrike cursor handling#44548
efd6 merged 3 commits intoelastic:mainfrom
efd6:44364-streaming_crowdstrike

Conversation

@efd6
Copy link
Contributor

@efd6 efd6 commented May 29, 2025

Proposed commit message

x-pack/filebeat/input/streaming: fix crowdstrike cursor handling

When the follower is given a non-singular set of resource descriptions,
it incorrectly uses the offset obtained from the registry across all of
the resources, and each of those resources' work loops writes their
cursor updates to the same (incorrectly) shared offset. This results in
cursor offset confusion.

The fix here is to retain cursors for each of the resources, keyed on
the dataFeedURL for each resource.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs
@efd6 efd6 self-assigned this May 29, 2025
@efd6 efd6 added Filebeat Filebeat bugfix Team:Security-Service Integrations Security Service Integrations Team backport-8.18 Automated backport to the 8.18 branch backport-9.0 Automated backport to the 9.0 branch backport-8.19 Automated backport to the 8.19 branch labels May 29, 2025
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 29, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 29, 2025

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
@efd6 efd6 force-pushed the 44364-streaming_crowdstrike branch from c717bfa to 1f5b338 Compare May 29, 2025 04:45
@mergify
Copy link
Contributor

mergify bot commented Jun 4, 2025

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 44364-streaming_crowdstrike upstream/44364-streaming_crowdstrike
git merge upstream/main
git push upstream 44364-streaming_crowdstrike
@efd6 efd6 force-pushed the 44364-streaming_crowdstrike branch from 1f5b338 to 32eaf5c Compare June 4, 2025 07:47
@mergify
Copy link
Contributor

mergify bot commented Jun 4, 2025

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 44364-streaming_crowdstrike upstream/44364-streaming_crowdstrike
git merge upstream/main
git push upstream 44364-streaming_crowdstrike
@efd6 efd6 force-pushed the 44364-streaming_crowdstrike branch 2 times, most recently from 3888c60 to 0d534c1 Compare June 5, 2025 05:35
@efd6 efd6 marked this pull request as ready for review June 10, 2025 06:57
@efd6 efd6 requested review from a team as code owners June 10, 2025 06:57
@elasticmachine
Copy link
Contributor

elasticmachine commented Jun 10, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
chemamartinez
chemamartinez approved these changes Jun 10, 2025
View reviewed changes
Copy link
Contributor

@chemamartinez chemamartinez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@mergify
Copy link
Contributor

mergify bot commented Jun 10, 2025

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 44364-streaming_crowdstrike upstream/44364-streaming_crowdstrike
git merge upstream/main
git push upstream 44364-streaming_crowdstrike
@efd6 efd6 force-pushed the 44364-streaming_crowdstrike branch from 8554d3b to 4674bdc Compare June 10, 2025 21:11
@efd6 efd6 requested a review from andrewkroh June 10, 2025 22:55
@mergify
Copy link
Contributor

mergify bot commented Jun 10, 2025

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 44364-streaming_crowdstrike upstream/44364-streaming_crowdstrike
git merge upstream/main
git push upstream 44364-streaming_crowdstrike
@efd6 efd6 force-pushed the 44364-streaming_crowdstrike branch from 4674bdc to e05f7de Compare June 10, 2025 23:02
ShourieG
ShourieG approved these changes Jun 11, 2025
View reviewed changes
Copy link
Contributor

@ShourieG ShourieG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@mergify
Copy link
Contributor

mergify bot commented Jun 12, 2025

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 44364-streaming_crowdstrike upstream/44364-streaming_crowdstrike
git merge upstream/main
git push upstream 44364-streaming_crowdstrike
@efd6
x-pack/filebeat/input/streaming: fix crowdstrike cursor handling
b14b651 
When the follower is given a non-singular set of resource descriptions,
it incorrectly uses the offset obtained from the registry across all of
the resources, and each of those resources' work loops writes their
cursor updates to the same (incorrectly) shared offset. This results in
cursor offset confusion.

The fix here is to retain cursors for each of the resources, keyed on
the dataFeedURL for each resource.
efd6 added 2 commits June 12, 2025 10:00
@efd6
add cursor exercise option for testing
774029d
@efd6
fix incorrect FeedURL name use
fcdc64a 
Previously we were mutating the name to include the query; keep the URL and use
the retained version for the feed name. Otherwise we have unbounded dead feed
cursors stored.
@efd6 efd6 force-pushed the 44364-streaming_crowdstrike branch from e05f7de to fcdc64a Compare June 12, 2025 00:30
@efd6 efd6 merged commit cdf2355 into elastic:main Jun 12, 2025
31 of 34 checks passed
mergify bot pushed a commit that referenced this pull request Jun 12, 2025
@efd6 @mergify
x-pack/filebeat/input/streaming: fix crowdstrike cursor handling (#44548
933ed93 
)

When the follower is given a non-singular set of resource descriptions,
it incorrectly uses the offset obtained from the registry across all of
the resources, and each of those resources' work loops writes their
cursor updates to the same (incorrectly) shared offset. This results in
cursor offset confusion.

The fix here is to retain cursors for each of the resources, keyed on
the dataFeedURL for each resource.

(cherry picked from commit cdf2355)

# Conflicts:
#	x-pack/filebeat/docs/inputs/input-streaming.asciidoc
@mergify mergify bot mentioned this pull request Jun 12, 2025
Merged
6 tasks
mergify bot pushed a commit that referenced this pull request Jun 12, 2025
@efd6 @mergify
x-pack/filebeat/input/streaming: fix crowdstrike cursor handling (#44548
b9d0dbb 
)

When the follower is given a non-singular set of resource descriptions,
it incorrectly uses the offset obtained from the registry across all of
the resources, and each of those resources' work loops writes their
cursor updates to the same (incorrectly) shared offset. This results in
cursor offset confusion.

The fix here is to retain cursors for each of the resources, keyed on
the dataFeedURL for each resource.

(cherry picked from commit cdf2355)

# Conflicts:
#	x-pack/filebeat/docs/inputs/input-streaming.asciidoc
mergify bot pushed a commit that referenced this pull request Jun 12, 2025
@efd6 @mergify
x-pack/filebeat/input/streaming: fix crowdstrike cursor handling (#44548
fd60e85 
)

When the follower is given a non-singular set of resource descriptions,
it incorrectly uses the offset obtained from the registry across all of
the resources, and each of those resources' work loops writes their
cursor updates to the same (incorrectly) shared offset. This results in
cursor offset confusion.

The fix here is to retain cursors for each of the resources, keyed on
the dataFeedURL for each resource.

(cherry picked from commit cdf2355)
This was referenced Jun 12, 2025
Merged
Merged
efd6 added a commit to efd6/integrations that referenced this pull request Jun 12, 2025
@efd6
crowdstrike: add support for multi-resource streams
b1943a4 
elastic/beats#44548 added support for multi-resource stream. This
updates the CEL program that processes the events provided by the stream
so that it is able to handle the new cursor structure. It is able to
distinguish old agents from the new multi-resource aware agents by the
presence of the feed field in state.

When the agent is upgrade from the old state form to the new form, it is
expected that the integration will recollect the existing data since the
cursors are not compatible with each other.
efd6 added a commit to efd6/integrations that referenced this pull request Jun 12, 2025
@efd6
crowdstrike: add support for multi-resource streams
063e6f2 
elastic/beats#44548 added support for multi-resource stream. This
updates the CEL program that processes the events provided by the stream
so that it is able to handle the new cursor structure. It is able to
distinguish old agents from the new multi-resource aware agents by the
presence of the feed field in state.

When the agent is upgrade from the old state form to the new form, it is
expected that the integration will recollect the existing data since the
cursors are not compatible with each other.
efd6 added a commit that referenced this pull request Jun 12, 2025
@mergify @efd6
x-pack/filebeat/input/streaming: fix crowdstrike cursor handling (#44548
2ed6a64 
) (#44768)

When the follower is given a non-singular set of resource descriptions,
it incorrectly uses the offset obtained from the registry across all of
the resources, and each of those resources' work loops writes their
cursor updates to the same (incorrectly) shared offset. This results in
cursor offset confusion.

The fix here is to retain cursors for each of the resources, keyed on
the dataFeedURL for each resource.

(cherry picked from commit cdf2355)

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
efd6 added a commit that referenced this pull request Jun 12, 2025
@mergify @efd6
[8.19](backport #44548) x-pack/filebeat/input/streaming: fix crowdstr…
63f9153 
…ike cursor handling (#44767)

* x-pack/filebeat/input/streaming: fix crowdstrike cursor handling (#44548)

When the follower is given a non-singular set of resource descriptions,
it incorrectly uses the offset obtained from the registry across all of
the resources, and each of those resources' work loops writes their
cursor updates to the same (incorrectly) shared offset. This results in
cursor offset confusion.

The fix here is to retain cursors for each of the resources, keyed on
the dataFeedURL for each resource.

(cherry picked from commit cdf2355)

# Conflicts:
#	x-pack/filebeat/docs/inputs/input-streaming.asciidoc

* revert doc change for conflict resolution

* reapply doc change to resolve conflict

---------

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
efd6 added a commit that referenced this pull request Jun 12, 2025
@mergify @efd6
[8.18](backport #44548) x-pack/filebeat/input/streaming: fix crowdstr…
85fc355 
…ike cursor handling (#44766)

* x-pack/filebeat/input/streaming: fix crowdstrike cursor handling (#44548)

When the follower is given a non-singular set of resource descriptions,
it incorrectly uses the offset obtained from the registry across all of
the resources, and each of those resources' work loops writes their
cursor updates to the same (incorrectly) shared offset. This results in
cursor offset confusion.

The fix here is to retain cursors for each of the resources, keyed on
the dataFeedURL for each resource.

(cherry picked from commit cdf2355)

# Conflicts:
#	x-pack/filebeat/docs/inputs/input-streaming.asciidoc

* revert doc change for conflict resolution

* reapply doc change to resolve conflict

---------

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
@andrewkroh andrewkroh mentioned this pull request Jun 12, 2025
Merged
5 tasks
efd6 added a commit to elastic/integrations that referenced this pull request Jun 15, 2025
@efd6
crowdstrike: add support for multi-resource streams (#14212)
b78b417 
elastic/beats#44548 added support for multi-resource stream. This
updates the CEL program that processes the events provided by the stream
so that it is able to handle the new cursor structure. It is able to
distinguish old agents from the new multi-resource aware agents by the
presence of the feed field in state.

When the agent is upgrade from the old state form to the new form, it is
expected that the integration will recollect the existing data since the
cursors are not compatible with each other.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-8.18 Automated backport to the 8.18 branch backport-8.19 Automated backport to the 8.19 branch backport-9.0 Automated backport to the 9.0 branch bugfix Filebeat Filebeat Team:Security-Service Integrations Security Service Integrations Team

6 participants

@efd6 @elasticmachine @andrewkroh @bmorelli25 @chemamartinez @ShourieG