[aws] Set region in config from GetBucketLocation API call

[kaiyan-sheng]

Proposed commit message

This PR sets the region of the S3 bucket as the region in AWS config to make API calls.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Without this change, run Filebeat aws-s3 input with this config and you will see error:

s3 GetObject failed: operation error S3: GetObject, https response error StatusCode: 301, RequestID: 01H7T2XWRZ6N44MW, HostID: WqYAZM+diCdCnBwj2i1tAzchSXBZql8SdIfQ0ZUCKHoR4YbMiBGD10duVYqIQ97pCg/j8MmJTgo=, api error PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

config:

filebeat.inputs:
- type: aws-s3
  enabled: true
  credential_profile_name: elastic-observability
  number_of_workers: 5
  bucket_arn: 'arn:aws:s3:::tkravchenko-test1'

The workaround is to add default_region: us-east-2 in the config.

With this change, Filebeat is able to get the bucket location/region from API call and assign it to the region config.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kaiyan-sheng? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b aws_region_s3 upstream/aws_region_s3
git merge upstream/main
git push upstream aws_region_s3

[Kavindu-Dodan]

➕ for using pointers. From first look, old code should have worked without any issue. But seems, there's some buggy optimization from Go, making the call at 28 to have same aws config copy as used in line 20 😁

Good work catching this and fixing 🙌

[tetianakravchenko]

Thank you for a quick fix!
I've tested filebeat and it works!