OPA Pulling OCI Bundle from AWS ECR

[vazquezf2000]

I am attempting to pull an OCI policy bundle from AWS' ECR service. When the request for the blob is made,

• OPA: 0.61.0
• AWS' ECR service returns a 307 to the location to actually download the blob,
• a GET request is then made from OPA to the location from the 307 response above,
• unfortunately, OPA is adding the Authorization header to the redirect request and AWS' S3 service is now rejecting because it only allows one auth mechanism

Please see the traffic flow below....

---

Fetch Request:

OPA Debug Log:
{
  "digest": "sha256:d739a15f0200147e662262ce6a2***************3b78f20a335831cddc",
  "level": "debug",
  "msg": "do request",
  "request.header.accept": "application/vnd.oci.image.layer.v1.tar+gzip, */*",
  "request.header.user-agent": "containerd/1.7.12+unknown",
  "request.method": "GET",
  "time": "2024-02-09T13:12:59-06:00",
  "url": "https://************.dkr.ecr.us-east-1.amazonaws.com/v2/c/t/at/opa-policy/blobs/sha256:d739a15f0200147e662262ce6a2***************3b78f20a335831cddc"
}

HTTP Dump:

GET /v2/c/t/at/opa-policy/blobs/sha256:d739a15f0200147e662262ce6a2***************3b78f20a335831cddc HTTP/1.1
Host: ************.dkr.ecr.us-east-1.amazonaws.com
User-Agent: containerd/1.7.12+unknown
Accept: application/vnd.oci.image.layer.v1.tar+gzip, */*
Authorization: Basic QVdTOmV5Snd****pNVFV4TVgwPQ==
Accept-Encoding: gzip

Fetch Response:

HTTP/1.1 307 Temporary Redirect
Date: Fri, 09 Feb 2024 19:12:33 GMT
Docker-Distribution-Api-Version: registry/2.0
Location: https://prod-us-east-1-starport-layer-bucket.s3.us-east-1.amazonaws.com/72d2de-*******-2cc31c9b-be0f-*****-794f-42b6dba897c0/e8ab7baa-****************-e0a77276e5b4?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEEMaCXVzLW****g27PuvbIcw%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240209T191233Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=ASIAYT*****R7%2F20240209%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=0b2fa002*******b377ac24b58d
Sizes: 
Content-Length: 0

Redirect Request:

GET /72d2de-*******-2cc31c9b-be0f-*****-794f-42b6dba897c0/e8ab7baa-****************-e0a77276e5b4?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEEMaCXVzLW****g27PuvbIcw%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240209T191233Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=ASIAYT*****R7%2F20240209%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=0b2fa002*******b377ac24b58d HTTP/1.1
Host: prod-us-east-1-starport-layer-bucket.s3.us-east-1.amazonaws.com
User-Agent: containerd/1.7.12+unknown
Accept: application/vnd.oci.image.layer.v1.tar+gzip, */*
Authorization: Basic QVdTOmV5Snd****pNVFV4TVgwPQ==
Referer: https://************.dkr.ecr.us-east-1.amazonaws.com/v2/c/t/at/opa-policy/blobs/sha256:d739a15f0200147e662262ce6a2***************3b78f20a335831cddc
Accept-Encoding: gzip

Redirect Response:

HTTP/1.1 400 Bad Request
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 09 Feb 2024 19:12:59 GMT
Server: AmazonS3
X-Amz-Id-2: iz7oHaost9LfM8PJjnubFtnNJ10XRB5N6Br3nkrvzr9486jv8j3rxdAH1P/fHeboozgGjP6PrUw=
X-Amz-Request-Id: Y9S2S96AX836GN40

e5b
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>Basic QVdTOmV5Snd****pNVFV4TVgwPQ==</ArgumentValue><RequestId>Y9S2S96AX836GN40</RequestId><HostId>iz7oHaost9LfM8PJjnubFtnNJ10XRB5N6Br3nkrvzr9486jv8j3rxdAH1P/fHeboozgGjP6PrUw=</HostId></Error>
0

OPA Debug Log:

{
  "digest": "sha256:d739a15f0200147e662262ce6a2***************3b78f20a335831cddc",
  "level": "debug",
  "msg": "fetch response received",
  "response.header.content-type": "application/xml",
  "response.header.date": "Fri, 09 Feb 2024 19:12:59 GMT",
  "response.header.server": "AmazonS3",
  "response.header.x-amz-id-2": "iz7oHaost9LfM8PJjnubFtnNJ10XRB5N6Br3nkrvzr9486jv8j3rxdAH1P/fHeboozgGjP6PrUw=",
  "response.header.x-amz-request-id": "Y9S2S96AX836GN40",
  "response.status": "400 Bad Request",
  "time": "2024-02-09T13:12:59-06:00",
  "url": "https://************.dkr.ecr.us-east-1.amazonaws.com/v2/c/t/at/opa-policy/blobs/sha256:d739a15f0200147e662262ce6a2***************3b78f20a335831cddc"
}

Steps To Reproduce

Followed instructions at:

https://www.openpolicyagent.org/docs/latest/configuration/#using-private-image-from-oci-repositories

created the following configuration:

services:
  aws-registry:
    url: https://*******.dkr.ecr.us-east-1.amazonaws.com/
    type: oci
    credentials:
      bearer:
        scheme: Basic
        token: AWS:eyJwYXlsb2FkIjoidmEwQm9u*********lwZSI6IkRBVEFfS0VZIiwiZXhwaXJhdGlvbiI6MTcwNzUzMTUxMX0=

bundles:
  authz:
    service: aws-registry
    resource: *******.dkr.ecr.us-east-1.amazonaws.com/c/t/at/opa-policy/opa-policy:DEV-1.0.0
    persist: true
    polling:
      min_delay_seconds: 10
      max_delay_seconds: 120

persistence_directory: /tmp/opa

default_decision: /corp/csi/authz/http/allow

decision_logs:
  console: true

caching:
  inter_query_builtin_cache:
    max_size_bytes: 10000000

addr: localhost:8181

Run via CLI as follows:

opa run -s --watch -c /temp/config.yaml --log-level=debug --log-format=json-pretty --diagnostic-addr=0.0.0.0:8282

Expected behavior

• Authorization header is not included in redirect calls,
• OCI policy bundle is downloaded (fetched) and setup correctly

Additional context

Please let me know if you need additional information :-)

[ashutosh-narkar]

@carabasdaniel @gitu any thoughts here?

[carabasdaniel]

Hi @vazquezf2000,

Thanks for the detailed issue, does oras pull work for that image ?

[vazquezf2000]

oras pull from the CLI works as expected and so does using the oras lib directly via oras.copy ... if interested I can additionally post the output from the CLI oras pull.

[carabasdaniel]

Hi @vazquezf2000, according to the last comment here #6590 you should be able to get this working using the AWS rest plugin for the client credentials or you can use a custom plugin to prevent attaching the Authorization header on redirection.

[stale]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days. Although currently inactive, the issue could still be considered and actively worked on in the future. More details about the use-case this issue attempts to address, the value provided by completing it or possible solutions to resolve it would help to prioritize the issue.

[ashutosh-narkar]

Fixed in #6728.