Skip to content

feat: add status reporting to S3#45748

Merged
joecompute merged 65 commits intoelastic:mainfrom
joecompute:feat/s3-status-reporting
Aug 20, 2025
Merged

feat: add status reporting to S3#45748
joecompute merged 65 commits intoelastic:mainfrom
joecompute:feat/s3-status-reporting

Conversation

@joecompute
Copy link
Contributor

@joecompute joecompute commented Aug 5, 2025 •
edited
Loading

Proposed commit message

Adds Status reporting for AWS s3 inputs (via SQS and without SQS), and moves the proxy state reporter from the awscloudwatch package out into a common package.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None - Status reports are utilized by the Agent. In standalone mode, logs are added in Debug level

How to test this PR locally

  • Run standalone filebeat with an S3 input (either via SQS or without SQS) & debug logs
  • Observe debug logs with S3 input status updated: or S3 via SQS input status updated: prefix for status changes

Startup and configuration

S3 via SQS: Starting, configuring, running messages

image

S3 without SQS: Startup and configuration, plus with failure/stop via expired token

image

More degraded scenarios

S3 via SQS: Expired token

image

S3 without SQS: Network fault

image

S3 via SQS: Network fault

image

Related issues
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 5, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Aug 5, 2025

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
@mergify
Copy link
Contributor

mergify bot commented Aug 5, 2025

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @joecompute? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
  • backport-active-all is the label that automatically backports to all active branches.
  • backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
  • backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.
@joecompute joecompute added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Aug 5, 2025
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 5, 2025
@joecompute joecompute added backport-8.19 Automated backport to the 8.19 branch backport-9.1 Automated backport to the 9.1 branch labels Aug 5, 2025
@joecompute
Copy link
Contributor Author

joecompute commented Aug 5, 2025

run docs-build
joecompute
joecompute commented Aug 6, 2025
View reviewed changes
@joecompute joecompute force-pushed the feat/s3-status-reporting branch 2 times, most recently from be527cc to af85abf Compare August 8, 2025 17:39
joecompute
joecompute commented Aug 8, 2025
View reviewed changes
joecompute
joecompute commented Aug 8, 2025
View reviewed changes
@joecompute joecompute marked this pull request as ready for review August 9, 2025 00:22
@joecompute joecompute requested review from a team as code owners August 9, 2025 00:22
@elasticmachine
Copy link
Contributor

elasticmachine commented Aug 9, 2025

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)
@joecompute joecompute changed the title feat: add status reporting to S3 (WIP) Aug 9, 2025
Kavindu-Dodan
Kavindu-Dodan reviewed Aug 11, 2025
View reviewed changes
Kavindu-Dodan
Kavindu-Dodan reviewed Aug 11, 2025
View reviewed changes
@joecompute joecompute force-pushed the feat/s3-status-reporting branch 2 times, most recently from 1511f9f to a3acf1c Compare August 12, 2025 18:04
Kavindu-Dodan
Kavindu-Dodan reviewed Aug 13, 2025
View reviewed changes
Copy link
Contributor

@Kavindu-Dodan Kavindu-Dodan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good 🙌 , I will defer my approval till we verify the concern of this comment https://github.com/elastic/beats/pull/45748/files#r2273745257
@joecompute
Fix constructors with new expected param
d432296
@joecompute
Disable linting issues
4afe2a6
@joecompute
Add update to running when no auth error but no messages
1c24503
@joecompute
Pass queue URL for degraded reporting
224565b
@joecompute
Add bucket arn when bucket nonexistent
afc98b2
@joecompute
Remove TODO comment
0441441
@joecompute
Remove logp.TestingSetup no-ops
ca23f72
@joecompute
Remove testingsetup no-op
8ae3e6d
@joecompute
Remove more logging testingsetup no-ops
a11f620
@joecompute
Add info on key and bucket during degraded status
76535cf
@joecompute
Update degraded message to be more informative
d5b25e8
@joecompute
Update mockStatusReporter to record status and msg
a879c0a
@joecompute
Remove unused import
87fcf41
@joecompute
Improve status update with bucket and key
d48746b
@joecompute
Update mockStatusReporter to behave like status reporter helper
3e87105
@joecompute
Refactor mock of StatusReporterHelper
dda332f
@joecompute
Add test on status updates
7a20415
@joecompute
Add verification on err msg passthrough to status
a6f8da9
@joecompute joecompute force-pushed the feat/s3-status-reporting branch from cf37ec7 to a6f8da9 Compare August 20, 2025 14:48
@joecompute joecompute enabled auto-merge (squash) August 20, 2025 14:51
@joecompute joecompute merged commit 3ec4a35 into elastic:main Aug 20, 2025
97 checks passed
mergify bot pushed a commit that referenced this pull request Aug 20, 2025
@joecompute @mergify
feat: add status reporting to S3 (#45748)
9a59957 
* Add health status reporting

* Move status reporter helper from CloudWatch status reporting to libbeat

* Enhance status reporter helper

* Fix linting issues, esp related to logging

(cherry picked from commit 3ec4a35)

# Conflicts:
#	x-pack/filebeat/input/awss3/input_benchmark_test.go
#	x-pack/filebeat/input/awss3/sqs_s3_event_test.go
#	x-pack/filebeat/input/awss3/sqs_test.go
mergify bot pushed a commit that referenced this pull request Aug 20, 2025
@joecompute @mergify
feat: add status reporting to S3 (#45748)
878b969 
* Add health status reporting

* Move status reporter helper from CloudWatch status reporting to libbeat

* Enhance status reporter helper

* Fix linting issues, esp related to logging

(cherry picked from commit 3ec4a35)
This was referenced Aug 20, 2025
Merged
Merged
kaisecheng pushed a commit to kaisecheng/beats that referenced this pull request Aug 20, 2025
@joecompute @kaisecheng
feat: add status reporting to S3 (elastic#45748)
0550cd9 
* Add health status reporting

* Move status reporter helper from CloudWatch status reporting to libbeat

* Enhance status reporter helper

* Fix linting issues, esp related to logging
joecompute pushed a commit that referenced this pull request Aug 21, 2025
@mergify
[8.19](backport #45748) feat: add status reporting to S3 (#46117)
fc1ecd0 
* Add health status reporting

* Move status reporter helper from CloudWatch status reporting to libbeat

* Enhance status reporter helper

* Fix linting issues, esp related to logging
joecompute pushed a commit that referenced this pull request Aug 21, 2025
@mergify
[9.1](backport #45748) feat: add status reporting to S3 (#46118)
2d68b5c 
* Add health status reporting

* Move status reporter helper from CloudWatch status reporting to libbeat

* Enhance status reporter helper

* Fix linting issues, esp related to logging
@joecompute joecompute mentioned this pull request Aug 25, 2025
Merged
6 tasks
@brijesh-elastic brijesh-elastic mentioned this pull request Oct 21, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-8.19 Automated backport to the 8.19 branch backport-9.1 Automated backport to the 9.1 branch Team:obs-ds-hosted-services Label for the Observability Hosted Services team

9 participants

@joecompute @elasticmachine @AndersonQ @cmacknz @Kavindu-Dodan @kaiyan-sheng @khushijain21 @taylor-swanson @bhapas