[s3exporter] 405 returned when using the S3 exporter cross AWS accounts

[dabcoder]

Component(s)

S3 exporter

What happened?

Description

We're noticing errors in the opentelemetry-collector pod logs when trying to use the S3 exporter cross AWS accounts. e.g. when trying to collect logs from pods running in an EKS cluster in a given AWS account and send those to an S3 bucket located in a different AWS account. The error:

"error": "WebIdentityErr: failed to retrieve credentials\ncaused by: SerializationError: failed to unmarshal error message\n\tstatus code: 405, request id: \ncaused by: UnmarshalError: failed to unmarshal error message
…
caused by: unknown error response tag.

In AWS, when with S3 access logging, we can see the below error:

...REST.POST.OBJECT .../ "POST /.../ HTTP/1.1" 405 MethodNotAllowed....

Yet I've double checked my AWS IAM role, policy and trust relationship, and they look correct (actually using a similar role and policy does work when we're in the same AWS account).

Steps to Reproduce

Use the below config:

config:
  exporters:
    awss3:
      s3uploader:
        endpoint: "https://<bucket-name>.s3.<region>.amazonaws.com/<folder>"
        region: "<region>"
        s3_prefix: "logs"
        s3_partition: "hour"
        role_arn: "arn:aws:iam::<aws-account-id>:role/<aws-iam-role-name>"

Expected Result

No 405 response, and logs should appear in the S3 bucket.

Actual Result

405 response, no logs in the bucket

Additional information

Considering the AWS Go SDK v1 deprecation next year, it might be worth switching to v2 if possible. Related discussion.

Collector version

opentelemetry-collector-contrib:0.97.0

Environment information

Environment

EKS

OpenTelemetry Collector configuration

config:
  exporters:
    awss3:
      s3uploader:
        endpoint: "https://<bucket-name>.s3.<region>.amazonaws.com/<folder>"
        region: "<region>"
        s3_prefix: "logs"
        s3_partition: "hour"
        role_arn: "arn:aws:iam::<aws-account-id>:role/<aws-iam-role-name>"

Log output

See above.

Additional context

None

[github-actions]

Pinging code owners for exporter/awss3: @atoulme @pdelewski. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/awss3: @atoulme @pdelewski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

Thanks for the report. Any precise reproduction steps are appreciated. We will move to aws SDK v2 for sure.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/awss3: @atoulme @pdelewski

See Adding Labels via Comments if you do not have permissions to add labels yourself.