[AWS] Skip s3 test events in s3-sqs filebeat input

[kaiyan-sheng]

Proposed commit message

This PR adds support to automatically skip s3:TestEvent messages in the AWS S3 input. When Amazon S3 sends test notifications after configuring event notifications, these messages are now detected and silently skipped without generating errors or processing attempts.

problem

When configuring S3 event notifications, Amazon S3 sends a test message to verify the notification configuration. These test messages use a different format than regular S3 event notifications:

{
   "Service":"Amazon S3",
   "Event":"s3:TestEvent",
   "Time":"2014-10-13T15:57:02.089Z",
   "Bucket":"amzn-s3-demo-bucket",
   "RequestId":"5582815E1AEA5ADF",
   "HostId":"8cLeGAmw098X5cv4Zkwcmo8vvZa3eH3eKxsPzbB9wrR+YstdA6Knx4Ip8EXAMPLE"
}

Unlike regular S3 event notifications that use a Records array structure, test events use a simplified format with direct fields. Previously, the input would attempt to process these test events, which could lead to:

• Unnecessary processing attempts
• Potential error messages in logs
• Confusion about missing Records fields

Solution

This PR adds a logic to identify and skip s3:TestEvent messages before any processing occurs. The solution handles test events in both direct S3→SQS and S3→SNS→SQS message flows.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

N/A

Related issues

• Relates ssi: too many failing packages in daily job integrations#15472

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kaiyan-sheng? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[efd6]

Thanks.

[Kavindu-Dodan]

LGTM and thanks for the improvement 🙌

[github-actions]

@Mergifyio backport 8.19 9.1 9.2

[mergify]

backport 8.19 9.1 9.2

✅ Backports have been created

Details

• #47705 [8.19](backport #47635) [AWS] Skip s3 test events in s3-sqs filebeat input has been created for branch 8.19
• #47706 [9.1](backport #47635) [AWS] Skip s3 test events in s3-sqs filebeat input has been created for branch 9.1
• #47707 [9.2](backport #47635) [AWS] Skip s3 test events in s3-sqs filebeat input has been created for branch 9.2