Helm OCI Registry with refresh token #8097

roitalpaz · 2022-01-05T09:17:59Z

Summary

Right now when using AWS ECR as an OCI Helm chart registry you need to manually refresh the token.

as the below said, you get the password as an input and not a command line.
https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#helm-chart-repositories

as you can see below, in AWS docs one always generate the password

aws ecr get-login-password \
     --region us-west-2 | helm registry login \
     --username AWS \
     --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com

https://docs.aws.amazon.com/AmazonECR/latest/userguide/push-oci-artifact.html

Motivation

I added the AWS ECR repo to argo with the password auto-generated that was autogenerated today, but the token expires tomorrow. The helm registry will stop working tomorrow.

Proposal

Over there is the code that takes a password. I wonder if it will be possible to give a command line to execute instead?

argo-cd/util/helm/cmd.go

Line 79 in 2770c69

func (c *Cmd) RegistryLogin(repo string, creds Creds) (string, error) {

The text was updated successfully, but these errors were encountered:

lvidarte · 2022-01-11T13:43:41Z

I'm facing the same issue:

$ argocd repo list -o yaml
- connectionState:
    attemptedAt: "2022-01-11T13:09:36Z"
    message: |-
      Unable to connect to repository: rpc error: code = Unknown desc = `helm registry login 891114520281.dkr.ecr.us-east-1.amazonaws.com --username ****** --password ******` failed exit status 1: WARNING: Using --password ****** the CLI is insecure. Use --password-stdin.
      Error: login attempt to https://891114520281.dkr.ecr.us-east-1.amazonaws.com/v2/ failed with status: 403 Forbidden
    status: Failed
  enableOCI: true
  name: common-charts
  repo: 891114520281.dkr.ecr.us-east-1.amazonaws.com
  type: helm
  username: AWS

helm version: v3.6.0
argocd: v2.2.1+122ecef
argocd-server: v2.1.2+7af9dfb

fokolo · 2022-01-12T09:13:59Z

Someone posted on slack this workaround:
https://github.com/smcavallo/argocd-ecr-updater

We are using ECR with Argocd and have found that we had to implement automation to refresh the tokens for the ECR repo before they expire. We are using this tool for the automation

amirb-argon · 2022-07-11T13:49:12Z

+1 on this.
This issue makes it very hard to adopt OCI helm repositories hosted on ECR + ArgoCD.
The use case behind this is to be able to lock ArgoCD apps to a specific in-house (private) helm chart when you want to do breaking changes.

Would really appreciate if someone could implement this natively in ArgoCD, similar to how it's implemented here:
https://argocd-image-updater.readthedocs.io/en/stable/configuration/registries/#credentials-caching

chrisduong · 2022-09-15T07:21:23Z

I had tried to use @fokolo tool but once I migrated my App to purely Helm, I faced "Manifest error cache hit", it remembered the old CMP, and don't run the helm registry login, hence it failed to download the chart from ECR.

And this cache is very hard clear, I delete all pods and use UI "Invalidate Cache" for the cluster, but it keep coming up.

CC: @wanghong230 , this is what happen, have this shed the light. source of discussion: https://cloud-native.slack.com/archives/C01TSERG0KZ/p1663165535172039

minhnnhat · 2022-09-20T03:26:27Z

You guys can give this cronjob a try, it renew both the ecr registry and ecr helm chart credentials:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ecr-credentials-sync
  namespace: argocd
rules:
- apiGroups: [""]
  resources:
  - secrets
  verbs:
  - get
  - create
  - patch
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ecr-credentials-sync
  namespace: argocd
subjects:
- kind: ServiceAccount
  name: ecr-credentials-sync
roleRef:
  kind: Role
  name: ecr-credentials-sync
  apiGroup: ""
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ecr-credentials-sync
  namespace: argocd
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: ecr-credentials-sync
  namespace: argocd
spec:
  schedule: "*/2 * * * *"
  failedJobsHistoryLimit: 1
  successfulJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: ecr-credentials-sync
          restartPolicy: Never
          volumes:
          - name: token
            emptyDir:
              medium: Memory
          initContainers:
          - image: amazon/aws-cli
            name: get-token
            imagePullPolicy: IfNotPresent
            env:
            - name: REGION
              value: ap-southeast-1 # change this if ECR repo is in a different region
            volumeMounts:
            - mountPath: /token
              name: token
            command:
            - /bin/sh
            - -ce
            - aws ecr get-login-password --region ${REGION} > /token/ecr-token
          containers:
          - image: bitnami/kubectl
            name: create-secret
            imagePullPolicy: IfNotPresent
            env:
            - name: SECRET_NAME
              value: ecr-credentials
            - name: ECR_REGISTRY
              value: <accountid>.dkr.ecr.<region>.amazonaws.com # fill in the account id and region
            volumeMounts:
            - mountPath: /token
              name: token
            command:
            - /bin/bash
            - -ce
            - |-
              kubectl -n argocd create secret docker-registry $SECRET_NAME \
                --dry-run=client \
                --docker-server="$ECR_REGISTRY" \
                --docker-username=AWS \
                --docker-password="$(</token/ecr-token)" \
                -o yaml | kubectl apply -f - && \
              cat <<EOF | kubectl apply -f -
              apiVersion: v1
              kind: Secret
              metadata:
                name: argocd-ecr-helm-credentials
                namespace: argocd
                labels:
                  argocd.argoproj.io/secret-type: repository
              stringData:
                username: AWS
                password: $(</token/ecr-token)
              EOF

chrisduong · 2022-09-20T08:52:59Z

I had tried to use @fokolo tool but once I migrated my App to purely Helm, I faced "Manifest error cache hit", it remembered the old CMP, and don't run the helm registry login, hence it failed to download the chart from ECR.

And this cache is very hard clear, I delete all pods and use UI "Invalidate Cache" for the cluster, but it keep coming up.

CC: @wanghong230 , this is what happen, have this shed the light. source of discussion: https://cloud-native.slack.com/archives/C01TSERG0KZ/p1663165535172039

My problem was complicated because the logging system didn't say anything about the App was not allowed to access to the Helm repo.

After I configured the Project to allow the Apps to access the repo (Chart Museum repo), I didn't have the "caching error hit" anymore ('helm pull failed - Authorisation problem')

I haven't tried with Helm OCI repo but I think it would work as well.

prein · 2022-10-17T17:06:54Z

@minhnnhat does it work for you without setting the enableOCI: "true" in the repository type secret?

minhnnhat · 2022-10-17T17:11:24Z

@minhnnhat does it work for you without setting the enableOCI: "true" in the repository type secret?

@prein I use ECR for storing both image and helm chart, and enableOCI: "true" is required

EladLeev · 2022-11-12T15:41:29Z

Thanks @minhnnhat, nice workaround.
For future ref - In my case I needed to change and add a few keys to make it work (type: helm, enableOCI: true, etc), so the quickest way to figure out what was missing on my version, was simply to add the creds once from the CLI -

argocd repo add <account>.dkr.ecr.<region>.amazonaws.com --type helm --name <name> --enable-oci --username AWS --password $(aws ecr get-login-password)

Check the created secret, and merge it to the cronjob solution.

lukonjun · 2022-11-14T10:47:13Z

Hey @minhnnhat also thanks from my side, got it working. Just wanted to share some insights I had during doing this

Might be really obvious, but I first did not get that repositories in Argo are represented via Kubernetes Secrets with the annotation argocd.argoproj.io/secret-type: repository
If you pass the Secret with option stringData you need to quote some string to make it work, otherwise the cronjob will fail

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: argocd-ecr-helm-credentials
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  username: "AWS"
  password: $(</token/ecr-token)
  enableOCI: "true"
  name: "test-helm-chart"
  type: "helm"
  url: "xxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com"
EOF

SmaineTF1 · 2022-11-30T14:48:43Z

Is the solution proposed by @minhnnhat , the one recommended for now please?
Because I used the yaml proposed here and it's only working temporarily.
I'll give a try to the cronjob option and let you know.

karlderkaefer · 2022-12-02T16:35:12Z

just wanted to share as alternative to cronjob, I have build small app that is able to refresh tokens in interval. I'm planning to write a article about it. Using external secrets sounds like the optimal solution, but I have no tested it yet. However this solution works for me until we have something better
https://github.com/karlderkaefer/argocd-ecr-updater
https://artifacthub.io/packages/helm/argocd-aws-ecr-updater/argocd-ecr-updater

Smana · 2022-12-02T17:16:19Z

based on both @minhnnhat and @lukonjun awswers here is the cronjob I use and that works like a charm:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ecr-credentials-sync
  namespace: argocd
rules:
- apiGroups: [""]
  resources:
  - secrets
  verbs:
  - get
  - create
  - patch
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ecr-credentials-sync
  namespace: argocd
subjects:
- kind: ServiceAccount
  name: ecr-credentials-sync
roleRef:
  kind: Role
  name: ecr-credentials-sync
  apiGroup: ""
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: ecr-credentials-sync
  namespace: argocd
spec:
  schedule: "*/10 * * * *"
  successfulJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          serviceAccountName: ecr-credentials-sync
          volumes:
            - emptyDir:
                medium: Memory
              name: token

          initContainers:
            - image: amazon/aws-cli
              name: get-token
              imagePullPolicy: IfNotPresent
              env:
                - name: REGION
                  value: <region> #!!! PUT YOUR AWS REGION HERE
              command:
                - /bin/sh
                - -ce
                - aws ecr get-login-password --region ${REGION} > /token/ecr-token
              volumeMounts:
                - mountPath: /token
                  name: token

          containers:
            - name: create-secret
              image: bitnami/kubectl
              imagePullPolicy: IfNotPresent
              env:
                - name: SECRET_NAME
                  value: ecr-credentials
                - name: ECR_REGISTRY
                  value: <account>.dkr.ecr.<region>.amazonaws.com #!!! PUT YOUR ECR REGISTRY HERE
              command:
                - /bin/bash
                - -ce
                - |-
                  kubectl -n argocd create secret docker-registry $SECRET_NAME \
                    --dry-run=client \
                    --docker-server="$ECR_REGISTRY" \
                    --docker-username=AWS \
                    --docker-password="$(</token/ecr-token)" \
                    -o yaml | kubectl apply -f - && \
                  cat <<EOF | kubectl apply -f -
                  apiVersion: v1
                  kind: Secret
                  metadata:
                    name: argocd-ecr-helm-credentials
                    namespace: argocd
                    labels:
                      argocd.argoproj.io/secret-type: repository
                  stringData:
                    username: AWS
                    password: $(</token/ecr-token)
                    enableOCI: "true"
                    name: "ECR"
                    type: "helm"
                    url: "<account>.dkr.ecr.<region>.amazonaws.com/helm-charts" #!!! PUT YOUR ECR URL HERE
                  EOF
              volumeMounts:
                - mountPath: /token
                  name: token

minhnnhat · 2022-12-09T07:12:17Z

based on both @minhnnhat and @lukonjun awswers here is the cronjob I use and that works like a charm:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ecr-credentials-sync
  namespace: argocd
rules:
- apiGroups: [""]
  resources:
  - secrets
  verbs:
  - get
  - create
  - patch
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ecr-credentials-sync
  namespace: argocd
subjects:
- kind: ServiceAccount
  name: ecr-credentials-sync
roleRef:
  kind: Role
  name: ecr-credentials-sync
  apiGroup: ""
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: ecr-credentials-sync
  namespace: argocd
spec:
  schedule: "*/10 * * * *"
  successfulJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          serviceAccountName: ecr-credentials-sync
          volumes:
            - emptyDir:
                medium: Memory
              name: token

          initContainers:
            - image: amazon/aws-cli
              name: get-token
              imagePullPolicy: IfNotPresent
              env:
                - name: REGION
                  value: <region> #!!! PUT YOUR AWS REGION HERE
              command:
                - /bin/sh
                - -ce
                - aws ecr get-login-password --region ${REGION} > /token/ecr-token
              volumeMounts:
                - mountPath: /token
                  name: token

          containers:
            - name: create-secret
              image: bitnami/kubectl
              imagePullPolicy: IfNotPresent
              env:
                - name: SECRET_NAME
                  value: ecr-credentials
                - name: ECR_REGISTRY
                  value: <account>.dkr.ecr.<region>.amazonaws.com #!!! PUT YOUR ECR REGISTRY HERE
              command:
                - /bin/bash
                - -ce
                - |-
                  kubectl -n argocd create secret docker-registry $SECRET_NAME \
                    --dry-run=client \
                    --docker-server="$ECR_REGISTRY" \
                    --docker-username=AWS \
                    --docker-password="$(</token/ecr-token)" \
                    -o yaml | kubectl apply -f - && \
                  cat <<EOF | kubectl apply -f -
                  apiVersion: v1
                  kind: Secret
                  metadata:
                    name: argocd-ecr-helm-credentials
                    namespace: argocd
                    labels:
                      argocd.argoproj.io/secret-type: repository
                  stringData:
                    username: AWS
                    password: $(</token/ecr-token)
                    enableOCI: "true"
                    name: "ECR"
                    type: "helm"
                    url: "<account>.dkr.ecr.<region>.amazonaws.com/helm-charts" #!!! PUT YOUR ECR URL HERE
                  EOF
              volumeMounts:
                - mountPath: /token
                  name: token

Thanks mate! That's work perfectly

tuananh · 2022-12-15T07:21:36Z

is this still the recommended way now? there should be better way to do this.

alexef · 2022-12-15T08:40:14Z

yes, the cronjob is the only way to deal with this for now.

However, see this other conversation about an alternative using ESO

useful for ECR OCI helm repositories which require credential refresh after 12 hours ✅ Closes: argoproj#8097 Signed-off-by: Isaac Gaskin <[email protected]>

morremeyer · 2023-01-26T15:45:06Z

I was able to successfully implement the CronJob from #8097 (comment).

However, when trying to deploy a helm chart from one of our git repositories that uses a chart from the ECR registry as dependency, it fails the helm dependency build step.

The Application looks like this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-application
  namespace: argocd
spec:
  project: default
  source:
    path: charts/wrapper
    repoURL: "https://github.com/org/repo"
    targetRevision: main
    helm:
      releaseName: test-charts
      valueFiles:
        - values/environment.yaml
  destination:
    namespace: namespace
    name: some-cluster

The Chart.yaml:

apiVersion: v2
name: wrapper
description: wrapper chart for dependency
type: application

version: 0.3.0

dependencies:
- name: dependency
  version: 0.3.0
  repository: oci://AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/charts

And the error is

Unable to save changes: application spec for test-application is invalid: InvalidSpecError: Unable to generate manifests in charts/wrapper: rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not download oci://${AWS_ACCOUNT_ID}.dkr.ecr.us-east-1.amazonaws.com/charts/dependency: pulling from host ${AWS_ACCOUNT_ID}.dkr.ecr.us-east-1.amazonaws.com failed with status code [manifests 0.3.0]: 401 Unauthorized

Question: Am I right with the assumption that the authentication only works for the configured helm repository? Meaning it will not work for a git repository that uses helm with a dependency in the same helm repository?

In that case, I assume we'll have to wait for ArgoCD 2.6 where we can do multi-source Applications (which will remove the need for the helm chart dependency in our use case).

ahaw023 · 2023-02-08T02:05:35Z

I would strongly recommend looking at this option: https://external-secrets.io/v0.7.2/guides/generator/

azelezni · 2023-02-15T13:52:08Z

I can verify this works with external-secrets generator
We're running external-secrets with IAM role with the following policy:

{
  "Statement": [
    {
      "Action": [
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetAuthorizationToken",
        "ecr:BatchGetImage",
        "ecr:BatchCheckLayerAvailability"
      ],
      "Effect": "Allow",
      "Resource": "*",
      "Sid": "ecrRead"
    }
  ],
  "Version": "2012-10-17"
}

Confiugre ECR generator and external secret (this is installed with Helm):

apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
metadata:
  name: ecr
spec:
  region: {{ .Values.aws.region }}
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: ecr-oci
spec:
  refreshInterval: 30m
  target:
    name: ecr-oci
    template:
      metadata:
        labels:
          argocd.argoproj.io/secret-type: repository
      data:
        name: ecr-oci
        type: helm
        enableOCI: 'true'
        url: {{ .Values.aws.accountId }}.dkr.ecr.{{ .Values.aws.region }}.amazonaws.com
        password: '{{ printf "{{ .password }}" }}'
        username: '{{ printf "{{ .username }}" }}'
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: ECRAuthorizationToken
          name: ecr

This works well with chart dependencies (Chart.yaml):

apiVersion: v2
name: my-chart
description: my-chart
type: application
version: 1.2.3
appVersion: 1.2.3
dependencies:
  - name: my-other-chart
    version: 5.6.7
    repository: oci://<ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com

cabrinha · 2023-07-18T15:15:03Z

Do these objects need to be created in the argocd namespace?

azelezni · 2023-07-18T15:23:49Z

Yes, same namespace as ArgoCD is running in

cabrinha · 2023-07-18T15:29:43Z

Yes, same namespace as ArgoCD is running in

Does ESO also need to be running in the ArgoCD namespace? When using JWT with IRSA auth for the ECR Token, it seems to be looking for the "external-secrets" service account in the argocd namespace, when its usually installed to a different namespace.

azelezni · 2023-07-18T15:34:20Z

Yes, same namespace as ArgoCD is running in

Does ESO also need to be running in the ArgoCD namespace? When using JWT with IRSA auth for the ECR Token, it seems to be looking for the "external-secrets" service account in the argocd namespace, when its usually installed to a different namespace.

External-secrets can be installed in any namespace (we are running it in kube-system).
Anyway this conversation is beyond the scope of this issue so let's not spam needlessly.

blakepettersson · 2023-09-30T09:29:17Z

This conversation overlaps with #10218, and can be further discussed there.

stijnbrouwers · 2023-12-20T11:26:50Z

FYI: I searched for a couple of hours and tried the cronjobs and ExternalSecret solutions.
None of them worked for me at first. The missing bit was the Application's "passCredentials" property as described here:
https://argo-cd.readthedocs.io/en/stable/user-guide/helm/#helm-pass-credentials

After adding the property to my applications, It then worked for OCI dependency charts with both the ExternalSecret and cronjob solutions.

tummalachervu · 2024-02-06T23:51:24Z

Someone posted on slack this workaround: https://github.com/smcavallo/argocd-ecr-updater

We are using ECR with Argocd and have found that we had to implement automation to refresh the tokens for the ECR repo before they expire. We are using this tool for the automation

Can you please help me to provide steps or sample files , i am new to this technology. I have installed argo-ecr-update through helm chart in my argocd repo. What should i do next

roitalpaz added the enhancement New feature or request label Jan 5, 2022

prein mentioned this issue Oct 18, 2022

Natively support cloud provider Authentication mechanisms for Helm Chart OCI registries #8952

Closed

igaskin mentioned this issue Dec 30, 2022

feat: 🎸 adding helm repo password refresh #11857

Closed

10 tasks

igaskin added a commit to igaskin/argo-cd that referenced this issue Jan 3, 2023

feat: 🎸 adding helm repo password refresh

be1355d

useful for ECR OCI helm repositories which require credential refresh after 12 hours ✅ Closes: argoproj#8097 Signed-off-by: Isaac Gaskin <[email protected]>

welderpb mentioned this issue Mar 29, 2023

HOME var overide travisghansen/argo-cd-helmfile#27

Open

blakepettersson closed this as not planned Won't fix, can't repro, duplicate, stale Sep 30, 2023

paul-civitas mentioned this issue Jun 18, 2024

Tagged helm chart values in values.yaml don't match image tag aws/karpenter-provider-aws#5415

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Helm OCI Registry with refresh token #8097

Helm OCI Registry with refresh token #8097

roitalpaz commented Jan 5, 2022

lvidarte commented Jan 11, 2022 •

edited

Loading

fokolo commented Jan 12, 2022

amirb-argon commented Jul 11, 2022

chrisduong commented Sep 15, 2022

minhnnhat commented Sep 20, 2022 •

edited

Loading

chrisduong commented Sep 20, 2022

prein commented Oct 17, 2022

minhnnhat commented Oct 17, 2022

EladLeev commented Nov 12, 2022 •

edited

Loading

lukonjun commented Nov 14, 2022

SmaineTF1 commented Nov 30, 2022

karlderkaefer commented Dec 2, 2022

Smana commented Dec 2, 2022 •

edited

Loading

minhnnhat commented Dec 9, 2022

tuananh commented Dec 15, 2022

alexef commented Dec 15, 2022

morremeyer commented Jan 26, 2023

ahaw023 commented Feb 8, 2023

azelezni commented Feb 15, 2023

cabrinha commented Jul 18, 2023

azelezni commented Jul 18, 2023

cabrinha commented Jul 18, 2023

azelezni commented Jul 18, 2023

blakepettersson commented Sep 30, 2023

stijnbrouwers commented Dec 20, 2023

tummalachervu commented Feb 6, 2024

Helm OCI Registry with refresh token #8097

Helm OCI Registry with refresh token #8097

Comments

roitalpaz commented Jan 5, 2022

Summary

Motivation

Proposal

lvidarte commented Jan 11, 2022 • edited Loading

fokolo commented Jan 12, 2022

amirb-argon commented Jul 11, 2022

chrisduong commented Sep 15, 2022

minhnnhat commented Sep 20, 2022 • edited Loading

chrisduong commented Sep 20, 2022

prein commented Oct 17, 2022

minhnnhat commented Oct 17, 2022

EladLeev commented Nov 12, 2022 • edited Loading

lukonjun commented Nov 14, 2022

SmaineTF1 commented Nov 30, 2022

karlderkaefer commented Dec 2, 2022

Smana commented Dec 2, 2022 • edited Loading

minhnnhat commented Dec 9, 2022

tuananh commented Dec 15, 2022

alexef commented Dec 15, 2022

morremeyer commented Jan 26, 2023

ahaw023 commented Feb 8, 2023

azelezni commented Feb 15, 2023

cabrinha commented Jul 18, 2023

azelezni commented Jul 18, 2023

cabrinha commented Jul 18, 2023

azelezni commented Jul 18, 2023

blakepettersson commented Sep 30, 2023

stijnbrouwers commented Dec 20, 2023

tummalachervu commented Feb 6, 2024

lvidarte commented Jan 11, 2022 •

edited

Loading

minhnnhat commented Sep 20, 2022 •

edited

Loading

EladLeev commented Nov 12, 2022 •

edited

Loading

Smana commented Dec 2, 2022 •

edited

Loading