fix: use aws http buildable client for add_cloud_metadata by Kavindu-Dodan · Pull Request #44189 · elastic/beats

Kavindu-Dodan · 2025-05-02T21:11:23Z

Proposed commit message

Use AWS-specific HTTP client awshttp.BuildableClient with overrides matching generic HTTP client of the processor. This allows the use of custom CA bundle loading and avoids failing internal to AWS SDK.

Fix was tested in an EC2 environment with AWS_CA_BUNDLE and it works as expected.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Fixes #44186

elasticmachine · 2025-05-05T17:33:28Z

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

Kavindu-Dodan · 2025-05-05T17:36:33Z

libbeat/processors/add_cloud_metadata/provider_aws_ec2.go

+	awsRegion := instanceIdentity.Region
+	accountID := instanceIdentity.AccountID
+	instanceID := instanceIdentity.InstanceID

-	_, _ = result.metadata.Put("cloud.instance.id", instanceIdentity.InstanceIdentityDocument.InstanceID)
-	_, _ = result.metadata.Put("cloud.machine.type", instanceIdentity.InstanceIdentityDocument.InstanceType)
+	_, _ = result.metadata.Put("cloud.instance.id", instanceIdentity.InstanceID)
+	_, _ = result.metadata.Put("cloud.machine.type", instanceIdentity.InstanceType)
 	_, _ = result.metadata.Put("cloud.region", awsRegion)
-	_, _ = result.metadata.Put("cloud.availability_zone", instanceIdentity.InstanceIdentityDocument.AvailabilityZone)
+	_, _ = result.metadata.Put("cloud.availability_zone", instanceIdentity.AvailabilityZone)
 	_, _ = result.metadata.Put("cloud.account.id", accountID)
-	_, _ = result.metadata.Put("cloud.image.id", instanceIdentity.InstanceIdentityDocument.ImageID)
+	_, _ = result.metadata.Put("cloud.image.id", instanceIdentity.ImageID)


Note - these are lint fixes and not related to core change.

kaiyan-sheng · 2025-05-05T18:11:59Z

libbeat/processors/add_cloud_metadata/provider_aws_ec2.go

+	// generate AWS specific client with overriding requirements
+	var awsHTTPClient awshttp.BuildableClient
+	awsHTTPClient = *awsHTTPClient.WithTimeout(client.Timeout)
+	awsHTTPClient = *awsHTTPClient.WithTransportOptions(func(tr *http.Transport) {


Do we need both awsHTTPClient.WithTimeout and awsHTTPClient.WithTransportOptions here?

Good point, I did this here to match the derived client to the client we parsed into this function. And this generic client is derived here [1] and have some overriding options :)

[1] - https://github.com/elastic/beats/blob/main/libbeat/processors/add_cloud_metadata/providers.go#L164-L175

Maybe something like this then?

awsHTTPClient := *awshttp.NewBuildableClient().WithTimeout(client.Timeout).WithTransportOptions(func(tr *http.Transport) { transport, ok := client.Transport.(*http.Transport) if ok { tr.TLSClientConfig = transport.TLSClientConfig } tr.DisableKeepAlives = true })

Yeah that's a nice formatting. I have adopted it - 14fb736 :)

zmoog · 2025-05-05T18:14:36Z

libbeat/processors/add_cloud_metadata/provider_aws_ec2.go

+			tr.TLSClientConfig = transport.TLSClientConfig
+		}
+
+		tr.DisableCompression = true


Why are we setting tr.DisableCompression = true?

ahh sorry, already updated this :) should have been DisableKeepAlives and matches with generic client. Also related to this discussion #44189 (comment)

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

zmoog

LGTM

* use aws http client Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review change : use chaining Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit f54c496)

* upstream/main: bk: use OIDC to create AWS cloud resources (elastic#44202) jenkins: remove references to the Jenkins pipelines and old packaging (elastic#41625) fix: use aws http buildable client for add_cloud_metadata (elastic#44189) [main](backport elastic#44166) docs: Prepare Changelog for 8.18.1 (elastic#44237)

…44248) * use aws http client * add changelog entry * review change : use chaining --------- (cherry picked from commit f54c496) Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com>

…44249) * use aws http client * add changelog entry * review change : use chaining --------- (cherry picked from commit f54c496) Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com>

…44250) * use aws http client * add changelog entry * review change : use chaining --------- (cherry picked from commit f54c496) Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com>

…44251) * use aws http client * add changelog entry * review change : use chaining --------- (cherry picked from commit f54c496) Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 2, 2025

mergify bot assigned Kavindu-Dodan May 2, 2025

Kavindu-Dodan added Team:obs-ds-hosted-services Label for the Observability Hosted Services team backport-8.x Automated backport to the 8.x branch with mergify labels May 2, 2025

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 2, 2025

elastic deleted a comment from mergify bot May 2, 2025

Kavindu-Dodan force-pushed the fix/change-aws-client branch 2 times, most recently from 867e446 to a18f8e2 Compare May 5, 2025 17:01

Kavindu-Dodan marked this pull request as ready for review May 5, 2025 17:33

Kavindu-Dodan requested a review from a team as a code owner May 5, 2025 17:33

Kavindu-Dodan force-pushed the fix/change-aws-client branch from a0d1ca0 to 4f68877 Compare May 5, 2025 17:35

Kavindu-Dodan commented May 5, 2025

View reviewed changes

Kavindu-Dodan changed the title ~~fix: use aws http client for add_cloud_metadata~~ May 5, 2025

kaiyan-sheng reviewed May 5, 2025

View reviewed changes

zmoog reviewed May 5, 2025

View reviewed changes

Kavindu-Dodan force-pushed the fix/change-aws-client branch from 4f68877 to 82aaa3d Compare May 5, 2025 18:21

Kavindu-Dodan requested review from kaiyan-sheng and zmoog May 5, 2025 18:27

Kavindu-Dodan added 3 commits May 5, 2025 13:14

use aws http client

b0f09cc

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

add changelog entry

d29ff53

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

review change : use chaining

ef860c7

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan force-pushed the fix/change-aws-client branch from 14fb736 to ef860c7 Compare May 5, 2025 20:14

kaiyan-sheng approved these changes May 5, 2025

View reviewed changes

zmoog approved these changes May 7, 2025

View reviewed changes

Kavindu-Dodan merged commit f54c496 into elastic:main May 7, 2025
195 checks passed

mergify bot mentioned this pull request May 7, 2025

[8.17](backport #44189) fix: use aws http buildable client for add_cloud_metadata #44248

Merged

6 tasks

mergify bot mentioned this pull request May 7, 2025

[8.18](backport #44189) fix: use aws http buildable client for add_cloud_metadata #44249

Merged

6 tasks

mergify bot mentioned this pull request May 7, 2025

[8.19](backport #44189) fix: use aws http buildable client for add_cloud_metadata #44250

Merged

6 tasks

mergify bot mentioned this pull request May 7, 2025

[9.0](backport #44189) fix: use aws http buildable client for add_cloud_metadata #44251

Merged

6 tasks

Kavindu-Dodan removed backport-active-8 Automated backport with mergify to all the active 8.[0-9]+ branches backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches labels May 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: use aws http buildable client for add_cloud_metadata#44189

fix: use aws http buildable client for add_cloud_metadata#44189
Kavindu-Dodan merged 3 commits intoelastic:mainfrom
Kavindu-Dodan:fix/change-aws-client

Kavindu-Dodan commented May 2, 2025 •

edited

Loading

elasticmachine commented May 5, 2025

Kavindu-Dodan May 5, 2025

kaiyan-sheng May 5, 2025

Kavindu-Dodan May 5, 2025

kaiyan-sheng May 5, 2025

Kavindu-Dodan May 5, 2025 •

edited

Loading

zmoog May 5, 2025

Kavindu-Dodan May 5, 2025

zmoog left a comment

Uh oh!

Labels

4 participants

Conversation

Kavindu-Dodan commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Related issues

elasticmachine commented May 5, 2025

Kavindu-Dodan May 5, 2025

Choose a reason for hiding this comment

kaiyan-sheng May 5, 2025

Choose a reason for hiding this comment

Kavindu-Dodan May 5, 2025

Choose a reason for hiding this comment

kaiyan-sheng May 5, 2025

Choose a reason for hiding this comment

Kavindu-Dodan May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

zmoog May 5, 2025

Choose a reason for hiding this comment

Kavindu-Dodan May 5, 2025

Choose a reason for hiding this comment

zmoog left a comment

Choose a reason for hiding this comment

Uh oh!

Labels

4 participants

Kavindu-Dodan commented May 2, 2025 •

edited

Loading

Kavindu-Dodan May 5, 2025 •

edited

Loading