Installing helm chart with CustomResources fails when installing other resources

[logileifs]

Hello!

• Vote on this issue by adding a 👍 reaction
• To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already)

Issue details

I'm trying to install argocd helm chart which contains some CustomResourceDefinitions and CustomResources. It works fine when I only install the helm chart and nothing else but as soon as I add something else like creating a few namespaces and secrets the helm chart installation fails because pulumi seems to be creating the CustomResources before the CustomResourceDefinitions have been installed.

Steps to reproduce

1. Install argocd helm chart, everything works fine
2. Add some more stuff like 8 namespaces and 2 secrets and installation of argocd chart fails

Expected: I expect pulumi to install CustomResourceDefinitions before it tries to install CustomResources when installing a helm chart
Actual: pulumi tries to install CustomResources before CustomResourceDefinitions have been installed

argo_ns = core.Namespace(
    'argocd-namespace',
    metadata={'name': 'argocd', 'labels': {'name': 'argocd'}},
    #opts=pulumi.ResourceOptions(provider=k8s_provider)
)
argo_chart = helm.Chart(
    'argocd',
    helm.ChartOpts(
        skip_await=True,
        namespace='argocd',
        values=values['argocd'],
        chart='argocd',
        #transformations=[],
        fetch_opts=helm.FetchOpts(
            password=config.require('helm_password'),
            repo=config.require('helm_repo'),
            username=config.require('helm_user')
        ),
    ),
    opts=pulumi.ResourceOptions(
        #provider=k8s_provider,
        depends_on=[argo_ns]
    )
)

This works fine and gives me the following output:

     Type                                                                    Name                                          Status       Info
 +   pulumi:pulumi:Stack                                                     brall-dev-dev                                 creating     2021/09/29 12:50:18 info: skipping unknown hook: "crd-install"
 +   ├─ kubernetes:core/v1:Namespace                                         argocd-namespace                              created
 +   └─ kubernetes:helm.sh/v3:Chart                                          argocd                                        creating
 +   └─ kubernetes:helm.sh/v3:Chart                                          argocd                                        creating.
 +      ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole               argocd-server                                 created
 +   └─ kubernetes:helm.sh/v3:Chart                                          argocd                                        creating..
 +   └─ kubernetes:helm.sh/v3:Chart                                          argocd                                        creating...
 +   └─ kubernetes:helm.sh/v3:Chart                                          argocd                                        creating
 +      ├─ kubernetes:apiextensions.k8s.io/v1beta1:CustomResourceDefinition  applications.argoproj.io                      created
 +      ├─ kubernetes:rbac.authorization.k8s.io/v1:Role                      argocd/argocd-application-controller          created
 +   └─ kubernetes:helm.sh/v3:Chart                                          argocd                                        creating.
 +      ├─ kubernetes:rbac.authorization.k8s.io/v1:Role                      argocd/argocd-dex-server                      created
 +      ├─ kubernetes:rbac.authorization.k8s.io/v1:RoleBinding               argocd/argocd-application-controller          created
 +      ├─ kubernetes:rbac.authorization.k8s.io/v1:RoleBinding               argocd/argocd-server                          created
 +      ├─ kubernetes:rbac.authorization.k8s.io/v1:RoleBinding               argocd/argocd-dex-server                      created
 +      ├─ kubernetes:networking.k8s.io/v1beta1:Ingress                      argocd/argocd-server                          created
 +      ├─ kubernetes:networking.k8s.io/v1:NetworkPolicy                     argocd/argocd-allow-dns                       created
 +      ├─ kubernetes:networking.k8s.io/v1:NetworkPolicy                     argocd/argocd-application-controller          created
 +      ├─ kubernetes:networking.k8s.io/v1:NetworkPolicy                     argocd/allow-argocd-communication             created
 +      ├─ kubernetes:networking.k8s.io/v1:NetworkPolicy                     argocd/argocd-repo-server                     created
 +      ├─ kubernetes:networking.k8s.io/v1:NetworkPolicy                     argocd/argocd-server                          created
 +      ├─ kubernetes:networking.k8s.io/v1:NetworkPolicy                     argocd/dex                                    created
 +      ├─ kubernetes:core/v1:ServiceAccount                                 argocd/argocd-application-controller          created
 +      ├─ kubernetes:core/v1:ServiceAccount                                 argocd/argocd-server                          created
 +      ├─ kubernetes:core/v1:ServiceAccount                                 argocd/argocd-dex-server                      created
 +      ├─ kubernetes:core/v1:Secret                                         argocd/argocd-secret                          created
 +      ├─ kubernetes:core/v1:ConfigMap                                      argocd/argocd-cm                              created
 +      ├─ kubernetes:core/v1:ConfigMap                                      argocd/argocd-rbac-cm                         created
 +      ├─ kubernetes:core/v1:ConfigMap                                      argocd/argocd-ssh-known-hosts-cm              created
 +      ├─ kubernetes:core/v1:ConfigMap                                      argocd/argocd-tls-certs-cm                    created
 +      ├─ kubernetes:core/v1:Service                                        argocd/argocd-application-controller-metrics  created
 +      ├─ kubernetes:core/v1:Service                                        argocd/argocd-application-controller          created
 +      ├─ kubernetes:core/v1:Service                                        argocd/argocd-repo-server-metrics             created
 +      ├─ kubernetes:core/v1:Service                                        argocd/argocd-repo-server                     created
 +      ├─ kubernetes:core/v1:Service                                        argocd/argocd-server-metrics                  created
 +      ├─ kubernetes:core/v1:Service                                        argocd/argocd-server                          created
 +      ├─ kubernetes:core/v1:Service                                        argocd/argocd-dex-server                      created
 +      ├─ kubernetes:core/v1:Service                                        argocd/argocd-redis                           created
 +      ├─ kubernetes:argoproj.io/v1alpha1:AppProject                        argocd/default                                created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/platform-support                       created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/transmutator                           created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/vault                                  created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/stevedore                              created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/portal                                 created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/workflow-service                       created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/argocd                                 created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/autoscaling                            created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/database                               created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/keycloak                               created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/traefik                                created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/monitoring                             created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/storage                                created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/gor                                    created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/cla                                    created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/phenocat                               created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/plat                                   created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/metering                               created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/sequence-miner                         created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/namespace                              created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/netpol-operator                        created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/smtp                                   created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/tertiarypipeline                       created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/datadog                                created
 +      ├─ kubernetes:argoproj.io/v1alpha1:Application                       argocd/custom-k8s-scheduler                   created
 +      ├─ kubernetes:monitoring.coreos.com/v1:PrometheusRule                argocd/argocd-alerts.rules                    created
 +      ├─ kubernetes:monitoring.coreos.com/v1:ServiceMonitor                argocd/argocd-application-controller          created
 +      ├─ kubernetes:monitoring.coreos.com/v1:ServiceMonitor                argocd/argocd-repo-server                     created
 +      ├─ kubernetes:monitoring.coreos.com/v1:ServiceMonitor                argocd/argocd-server                          created
 +      ├─ kubernetes:apps/v1:Deployment                                     argocd/argocd-application-controller          created
 +      ├─ kubernetes:apps/v1:Deployment                                     argocd/argocd-repo-server                     created
 +      ├─ kubernetes:apps/v1:Deployment                                     argocd/argocd-server                          created
 +      ├─ kubernetes:apps/v1:Deployment                                     argocd/argocd-dex-server                      created
 +      └─ kubernetes:apps/v1:Deployment                                     argocd/argocd-redis                           created

But the same code with 7 more namespaces and 2 secrets does not and gives me the following output:

     Type                                               Name                                          Status                  Info
 +   pulumi:pulumi:Stack                                brall-dev-dev                                 **creating failed**     1 error; 37 messages
 +   ├─ kubernetes:core/v1:Namespace                    argocd-namespace                              created
 +   ├─ kubernetes:core/v1:Namespace                    cla-namespace                                 created
 +   ├─ kubernetes:core/v1:Namespace                    database-namespace                            created
 +   ├─ kubernetes:core/v1:Namespace                    datadog-namespace                             created
 +   ├─ kubernetes:core/v1:Namespace                    keycloak-namespace                            created
 +   ├─ kubernetes:core/v1:Namespace                    monitoring-namespace                          created
 +   ├─ kubernetes:core/v1:Namespace                    workflows-namespace                           created
 +   ├─ kubernetes:core/v1:Namespace                    gorkube-namespace                             created
 +   ├─ kubernetes:helm.sh/v3:Chart                     argocd                                        created
 +   │  ├─ kubernetes:core/v1:ServiceAccount            argocd/argocd-application-controller          created
 +   │  ├─ kubernetes:core/v1:ServiceAccount            argocd/argocd-server                          created
 +   │  ├─ kubernetes:core/v1:ConfigMap                 argocd/argocd-rbac-cm                         created
 +   │  ├─ kubernetes:core/v1:Service                   argocd/argocd-application-controller-metrics  created
 +   │  ├─ kubernetes:core/v1:Service                   argocd/argocd-application-controller          created
 +   │  ├─ kubernetes:core/v1:ConfigMap                 argocd/argocd-ssh-known-hosts-cm              created
 +   │  ├─ kubernetes:core/v1:ConfigMap                 argocd/argocd-tls-certs-cm                    created
 +   │  ├─ kubernetes:core/v1:Service                   argocd/argocd-repo-server                     created
 +   │  ├─ kubernetes:core/v1:ConfigMap                 argocd/argocd-cm                              created
 +   │  ├─ kubernetes:core/v1:ServiceAccount            argocd/argocd-dex-server                      created
 +   │  ├─ kubernetes:core/v1:Service                   argocd/argocd-dex-server                      created
 +   │  ├─ kubernetes:core/v1:Service                   argocd/argocd-server-metrics                  created
 +   │  ├─ kubernetes:core/v1:Service                   argocd/argocd-repo-server-metrics             created
 +   │  ├─ kubernetes:core/v1:Service                   argocd/argocd-server                          created
 +   │  ├─ kubernetes:core/v1:Service                   argocd/argocd-redis                           created
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:AppProject   argocd/default                                **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/platform-support                       **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/transmutator                           **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/vault                                  **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/stevedore                              **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/portal                                 **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/workflow-service                       **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/argocd                                 **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/autoscaling                            **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/database                               **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/keycloak                               **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/traefik                                **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/monitoring                             **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/storage                                **creating failed**     1 error
 +   │  ├─ kubernetes:argoproj.io/v1alpha1:Application  argocd/gor                                    **creating failed**     1 error
 +   │  └─ kubernetes:argoproj.io/v1alpha1:Application  argocd/cla                                    **creating failed**     1 error
 +   ├─ kubernetes:core/v1:Secret                       datadog-api-key                               created
 +   └─ kubernetes:core/v1:Secret                       gorkube-dockerhub-credentials                 created

Diagnostics:
  kubernetes:argoproj.io/v1alpha1:Application (argocd/argocd):
    error: creation of resource argocd/argocd failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "Application" in version "argoproj.io/v1alpha1"

  kubernetes:argoproj.io/v1alpha1:Application (argocd/storage):
    error: creation of resource argocd/storage failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "Application" in version "argoproj.io/v1alpha1"

  kubernetes:argoproj.io/v1alpha1:Application (argocd/database):
    error: creation of resource argocd/database failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "Application" in version "argoproj.io/v1alpha1"

  kubernetes:argoproj.io/v1alpha1:Application (argocd/keycloak):
    error: creation of resource argocd/keycloak failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "Application" in version "argoproj.io/v1alpha1"

  kubernetes:argoproj.io/v1alpha1:Application (argocd/transmutator):
    error: creation of resource argocd/transmutator failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "Application" in version "argoproj.io/v1alpha1"

  kubernetes:argoproj.io/v1alpha1:Application (argocd/workflow-service):
    error: creation of resource argocd/workflow-service failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "Application" in version "argoproj.io/v1alpha1"

  kubernetes:argoproj.io/v1alpha1:Application (argocd/vault):
    error: creation of resource argocd/vault failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "Application" in version "argoproj.io/v1alpha1"

  kubernetes:argoproj.io/v1alpha1:Application (argocd/portal):
    error: creation of resource argocd/portal failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "Application" in version "argoproj.io/v1alpha1"

  kubernetes:argoproj.io/v1alpha1:Application (argocd/cla):
    error: creation of resource argocd/cla failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "Application" in version "argoproj.io/v1alpha1"

  kubernetes:argoproj.io/v1alpha1:Application (argocd/platform-support):
    error: creation of resource argocd/platform-support failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "Application" in version "argoproj.io/v1alpha1"

  kubernetes:argoproj.io/v1alpha1:Application (argocd/stevedore):
    error: creation of resource argocd/stevedore failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "Application" in version "argoproj.io/v1alpha1"

  kubernetes:argoproj.io/v1alpha1:Application (argocd/autoscaling):
    error: creation of resource argocd/autoscaling failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "Application" in version "argoproj.io/v1alpha1"

  kubernetes:argoproj.io/v1alpha1:Application (argocd/monitoring):
    error: creation of resource argocd/monitoring failed because the Kubernetes API server reported that the apiVersion for this resource does not exist. Verify that any required CRDs have been created: no matches for kind "Application" in version "argoproj.io/v1alpha1"

[logileifs]

This is similar to #1023 but since that one is closed and I have definite repro I decided to create another issue for it

[logileifs]

I tried adding all the namespaces and secrets as dependencies for the argo chart in the hope that it would install everything before argo and it would work the same as when I'm only installing the argo chart but that had no effect

[mikhailshilkov]

@lblackstone @viveklak any hints here?

[lblackstone]

Normally, the Chart will retry resource creation, but I'm guessing the skipAwait might be interfering with that. Can you try without that, and see if that works?

[logileifs]

@lblackstone I already tried without skip_await but the same happens

It does try the recreation up to 5 times at least but they all fail with the same result

[lblackstone]

Hmm, it sounds like it's not waiting long enough for the CRDs then. A couple more suggestions would be:

1. See if you can pull the CRD installation into a separate step to make sure they are ready before installing the chart
2. Try using the new helm.Release resource for the installation. (Note that this is still in developer preview, so probably not a good choice for anything mission critical just yet)

[logileifs]

I tried using the new helm.Release but that does not seem to work with a private helm repo.

argo_args = helm.ReleaseArgs(
    chart="argocd",
    skip_await=False,
    namespace='argocd',
    values=values['argocd'],
    repository_opts=helm.RepositoryOptsArgs(
        repo=config.require('helm_repo'),
        username=config.require('helm_user'),
        password=config.require('helm_password')
    )
)
argo_release = helm.Release("argocd", args=argo_args)

This gives me the following error:

error: failed to pull chart: looks like "https://genuitysci.jfrog.io/artifactory/gorkubbe" is not a valid chart repository or cannot be reached: failed to fetch https://genuitysci.jfrog.io/artifactory/gorkubbe/index.yaml : 401 Unauthorized

This is using the same credentials that work with helm.Chart so they are not the problem
I guess I am stuck with helm.Chart for now and trying to write some clever transformations to get around this issue

[twoTimesAgnew]

We are also intermittently having this error.
It can happen to us with any of our charts, but seems to happen a lot with istio-base or cert-manager. Most often, it will happen on the first pulumi up of a new cluster. However, it does sometimes happen on an existing cluster as well.

Has anyone had any luck fixing this or finding the source of the problem?

[ebr]

Running into this issue as well. Brand new cluster, trying to bootstrap ArgoCD using Pulumi. None of the custom resources can be created because the CRDs are not installed. I am using the helm.Release resource. Pulumi v3.94.2 Python SDK

[EronWright]

The essential problem here is that all the chart's child resources are created in parallel, to the limit set by the program's --parallel flag. There's an inherent race between the CRD and the CRs, and when there's a lot of resources, the CRD is less likely to be created early enough to avoid any errors.

The upcoming "Chart v4" resource will address this problem in two ways:

1. The Chart resource automatically adds dependency links between CRs and their CRD, so that the latter is installed first.
2. Taking a dependency on the chart itself is now an effective way to wait for all the chart resources.

I'm closing this issue as fixed in Chart v4.