feat: Introduce lastSync start position to AWS CloudWatch input backed by state registry by Kavindu-Dodan · Pull Request #43251 · elastic/beats

Kavindu-Dodan · 2025-03-12T20:56:07Z

Proposed commit message

Introduce lastSync option to AWS CloudWatch input's start_position configuration. This feature utilizes the state registry, which with this update stores the last successful sync timestamp under the last_sync_epoch key.

start_position configuration by default uses beginning option. This can cause data duplications when restarting filebeats or migrating it to a newer version. Users can avoid such data duplocates by utilizing lastSync option for the start_position configuration while targeting beats to use the same registry location.

Acknowledgement backed state updates

After the initial review, the need for acknowledgments prior to the state registry update was identified thanks to this remark - #43251 (comment)

Based on this feedback, I have implemented the state registry update with following component interactions,

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None - start_position by default use beginning. So this new option is an opt-in feature.

Screenshots

Example filebeat.yml configuration,

Example state entries with last sync timestamp updates,

elasticmachine · 2025-03-12T20:56:10Z

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

x-pack/filebeat/input/awscloudwatch/cloudwatch.go

MichaelKatsoulis

LGTM.
Very good work! I have only left some small comments.

elasticmachine · 2025-03-13T15:53:48Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

x-pack/filebeat/input/awscloudwatch/config.go

x-pack/filebeat/input/awscloudwatch/cloudwatch.go

x-pack/filebeat/input/awscloudwatch/state_handler.go

VihasMakwana

LGTM, @MichaelKatsoulis has already given his approval and code looks good.

faec

Thanks, looks good but one last very small but important change

x-pack/filebeat/input/awscloudwatch/cloudwatch_worker.go

faec

Thanks for all your work on this! Approved.

Kavindu-Dodan · 2025-06-06T15:16:59Z

Thanks for all your work on this! Approved.

Thank you @faec for the detailed review 🙏

Kavindu-Dodan · 2025-06-06T15:30:39Z

@elastic/beats-tech-leads - can I get a final approval :)

ebeahan

@Kavindu-Dodan question about the backporting label. If this is a new feature, should we only target landing in 8.19 & 9.1 instead of backport-active-all?

Kavindu-Dodan · 2025-06-09T15:10:19Z

@Kavindu-Dodan question about the backporting label. If this is a new feature, should we only target landing in 8.19 & 9.1 instead of backport-active-all?

Good point, plan was to merge only the necessary 9.1.x & 8.19.x and reject others. But I think that adds confusion so I will add only the needed :)

update - I have updated the labels :)

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> finalize AWS CW lastSync start position usage Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> update changelog next Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> review changes Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> review changes - improve state store id, fix timestamp and return nil for error Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> documentation update with md migration Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # x-pack/filebeat/input/awscloudwatch/config.go

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

…n handling Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

…d by state registry (#43251) * add state registry support for cloudwatch input Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> finalize AWS CW lastSync start position usage Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> update changelog next Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> review changes Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> review changes - improve state store id, fix timestamp and return nil for error Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> documentation update with md migration Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # x-pack/filebeat/input/awscloudwatch/config.go * review change: use ackers for confirmed delivery prior to saving state Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review: documentation updates Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes : ctx handling, shutdown improvement and race condition handling Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review : fix ctx condition Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 42e50cf) # Conflicts: # docs/reference/filebeat/filebeat-input-aws-cloudwatch.md

…S CloudWatch input backed by state registry (#44736) * feat: Introduce lastSync start position to AWS CloudWatch input backed by state registry (#43251) * add state registry support for cloudwatch input Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> finalize AWS CW lastSync start position usage Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> update changelog next Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> review changes Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> review changes - improve state store id, fix timestamp and return nil for error Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> documentation update with md migration Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # x-pack/filebeat/input/awscloudwatch/config.go * review change: use ackers for confirmed delivery prior to saving state Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review: documentation updates Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes : ctx handling, shutdown improvement and race condition handling Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review : fix ctx condition Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 42e50cf) # Conflicts: # docs/reference/filebeat/filebeat-input-aws-cloudwatch.md * Update CHANGELOG.next.asciidoc * update CloudWatch documentation for backport Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com> Co-authored-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan added Team:obs-ds-hosted-services Label for the Observability Hosted Services team backport-active-all Automated backport with mergify to all the active branches labels Mar 12, 2025

Kavindu-Dodan self-assigned this Mar 12, 2025

Kavindu-Dodan requested review from a team as code owners March 12, 2025 20:56

Kavindu-Dodan requested review from VihasMakwana and rdner March 12, 2025 20:56

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Mar 12, 2025

Kavindu-Dodan changed the title ~~feat: Introduce lastSync start position to AWS CloudWatch input backed by registry~~ Mar 12, 2025

Kavindu-Dodan force-pushed the feat/registry-for-cloudwatch branch 2 times, most recently from 91bfda9 to fecf38d Compare March 12, 2025 21:26

MichaelKatsoulis reviewed Mar 13, 2025

View reviewed changes

x-pack/filebeat/input/awscloudwatch/cloudwatch.go Show resolved Hide resolved

MichaelKatsoulis reviewed Mar 13, 2025

View reviewed changes

x-pack/filebeat/input/awscloudwatch/cloudwatch.go Outdated Show resolved Hide resolved

MichaelKatsoulis reviewed Mar 13, 2025

View reviewed changes

x-pack/filebeat/input/awscloudwatch/cloudwatch.go Outdated Show resolved Hide resolved

MichaelKatsoulis approved these changes Mar 13, 2025

View reviewed changes

Kavindu-Dodan force-pushed the feat/registry-for-cloudwatch branch from 86c2166 to 1cdaf6a Compare March 13, 2025 14:27

pierrehilbert added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Mar 13, 2025

pierrehilbert requested a review from faec March 13, 2025 15:53

faec requested changes Mar 13, 2025

View reviewed changes

Kavindu-Dodan force-pushed the feat/registry-for-cloudwatch branch 2 times, most recently from 6507477 to 6eefa92 Compare March 13, 2025 22:02

Kavindu-Dodan requested a review from faec March 13, 2025 22:07

VihasMakwana approved these changes Mar 14, 2025

View reviewed changes

Kavindu-Dodan force-pushed the feat/registry-for-cloudwatch branch from 6eefa92 to d064c6e Compare March 17, 2025 17:29

Kavindu-Dodan requested a review from a team March 17, 2025 19:48

rdner removed their request for review March 19, 2025 12:44

Kavindu-Dodan marked this pull request as draft March 21, 2025 21:50

Kavindu-Dodan requested a review from faec June 3, 2025 19:34

faec reviewed Jun 6, 2025

View reviewed changes

x-pack/filebeat/input/awscloudwatch/cloudwatch_worker.go Outdated Show resolved Hide resolved

Kavindu-Dodan requested a review from faec June 6, 2025 13:39

faec approved these changes Jun 6, 2025

View reviewed changes

Kavindu-Dodan added backport-active-all Automated backport with mergify to all the active branches and removed backport-active-all Automated backport with mergify to all the active branches labels Jun 6, 2025

ebeahan reviewed Jun 9, 2025

View reviewed changes

Kavindu-Dodan added backport-8.19 Automated backport to the 8.19 branch and removed backport-active-all Automated backport with mergify to all the active branches labels Jun 9, 2025

Kavindu-Dodan requested a review from ebeahan June 9, 2025 15:12

Kavindu-Dodan added 5 commits June 10, 2025 06:36

review change: use ackers for confirmed delivery prior to saving state

289da18

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

review: documentation updates

3b835b9

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

review changes : ctx handling, shutdown improvement and race conditio…

85dd156

…n handling Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

review : fix ctx condition

4ab81a4

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan force-pushed the feat/registry-for-cloudwatch branch from 5782ba9 to 4ab81a4 Compare June 10, 2025 13:36

elastic deleted a comment from mergify bot Jun 10, 2025

Kavindu-Dodan requested a review from a team June 10, 2025 15:44

ebeahan approved these changes Jun 10, 2025

View reviewed changes

Kavindu-Dodan merged commit 42e50cf into elastic:main Jun 10, 2025
203 checks passed

mergify bot mentioned this pull request Jun 10, 2025

[8.19](backport #43251) feat: Introduce lastSync start position to AWS CloudWatch input backed by state registry #44736

Merged

6 tasks

vishaangelova mentioned this pull request Jul 28, 2025

[Docs] Add applies_to to changes from #43251 #45579

Merged

mergify bot mentioned this pull request Jul 28, 2025

[9.1](backport #45579) [Docs] Add applies_to to changes from #43251 #45598

Merged

Kavindu-Dodan mentioned this pull request Jul 29, 2025

[AWS] [CloudWatch] Improve description to highlight lastSync option elastic/integrations#14733

Merged

5 tasks

nt-ishida-y mentioned this pull request Sep 25, 2025

Make aws-cloudwatch input to ensure at-least-once delivery #32641

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Introduce lastSync start position to AWS CloudWatch input backed by state registry#43251

feat: Introduce lastSync start position to AWS CloudWatch input backed by state registry#43251
Kavindu-Dodan merged 5 commits intoelastic:mainfrom
Kavindu-Dodan:feat/registry-for-cloudwatch

Kavindu-Dodan commented Mar 12, 2025 •

edited

Loading

elasticmachine commented Mar 12, 2025

Uh oh!

Uh oh!

Uh oh!

MichaelKatsoulis left a comment

elasticmachine commented Mar 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

VihasMakwana left a comment

faec left a comment

Uh oh!

faec left a comment

Kavindu-Dodan commented Jun 6, 2025

Kavindu-Dodan commented Jun 6, 2025

ebeahan left a comment

Kavindu-Dodan commented Jun 9, 2025 •

edited

Loading

Uh oh!

Labels

10 participants

Conversation

Kavindu-Dodan commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Acknowledgement backed state updates

Checklist

Disruptive User Impact

Screenshots

elasticmachine commented Mar 12, 2025

Uh oh!

Uh oh!

Uh oh!

MichaelKatsoulis left a comment

Choose a reason for hiding this comment

elasticmachine commented Mar 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

VihasMakwana left a comment

Choose a reason for hiding this comment

faec left a comment

Choose a reason for hiding this comment

Uh oh!

faec left a comment

Choose a reason for hiding this comment

Kavindu-Dodan commented Jun 6, 2025

Kavindu-Dodan commented Jun 6, 2025

ebeahan left a comment

Choose a reason for hiding this comment

Kavindu-Dodan commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Labels

10 participants

Kavindu-Dodan commented Mar 12, 2025 •

edited

Loading

Kavindu-Dodan commented Jun 9, 2025 •

edited

Loading