Describe the bug
After upgrading Tempo from 2.2.1to 2.2.2 our Tempo instances could no longer access S3. We use AWS IRSA ¹ to provide AWS credentials.

The regression seems to be introduced by #2871
My current guess is the AWS SDK now prefers the EC2 instance role over IRSA.

To Reproduce
Steps to reproduce the behavior:

1. Deploy Tempo 2.2.1 on AWS EKS with S3 storage and IRSA authentication
2. Tempo should start and be able to access S3 as expected
3. Upgrade Tempo to 2.2.2
4. Tempo should now crash during startup with S3 permission errors

Expected behavior
Tempo should continue to use the AWS IRSA provided credentials.

Environment:

• Infrastructure: Kubernetes (AWS EKS)
• Deployment tool: Helm (tempo-distributed - 1.6.3 - upgraded from 1.6.2)

Additional Context
Service logs (compactor):

level=info ts=2023-09-04T09:01:59.662782222Z caller=main.go:221 msg="initialising OpenTracing tracer"
level=info ts=2023-09-04T09:01:59.680278556Z caller=main.go:108 msg="Starting Tempo" version="(version=2.2.2, branch=HEAD, revision=5e18d78ba)"
level=error ts=2023-09-04T09:01:59.728222984Z caller=main.go:111 msg="error running Tempo" err="failed to init module services error initialising module: store: failed to create store unexpected error from ListObjects on <bucket_name>: Access Denied"

AWS environment variables in a running container:

$ env | grep -i aws
AWS_ROLE_ARN=arn:aws:iam::XXXXXXXXXXX:role/eks-development-v2-tempo
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_STS_REGIONAL_ENDPOINTS=regional
AWS_DEFAULT_REGION=eu-central-1
AWS_REGION=eu-central-1

Storage configuration (copied from tempo-distributed Helm values):

    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: "arn:aws:iam::XXXXXXXXXXX:role/eks-development-v2-tempo"

    storage:
      trace:
        backend: s3
        s3:
          bucket: <bucket_name>
          endpoint: s3.eu-central-1.amazonaws.com
          region: eu-central-1 # Added during troubleshooting