Unable to deploy to GKE AutoPilot

[maelp]

What happened?

I'm adding a CertManager controller to my Kubernetes configuration, but when trying to deploy it I receive this error:

  kubernetes:helm.sh/v3:Release (cert-manager-helm):
    warning: Helm release "cert-manager-helm-eb578c1e" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then retry. Reason: failed post-install: timed out waiting for the condition
    error: 1 error occurred:
    	* Helm release "cert-manager/cert-manager-helm-eb578c1e" was created, but failed to initialize completely. Use Helm CLI to investigate.: failed to become available within allocated timeout. Error: Helm Release cert-manager/cert-manager-helm-eb578c1e: failed post-install: timed out waiting for the condition

Expected Behavior

I expect the controller to be properly installed

Steps to reproduce

This is the configuration I'm using

cert_manager_ns_name = "cert-manager"
cert_manager_ns = Namespace("cert-manager", metadata={"name": cert_manager_ns_name})
cert_manager_controller = pulumi_kubernetes_cert_manager.CertManager(
    "cert-manager",
    install_crds=True,
    helm_options=pulumi_kubernetes_cert_manager.ReleaseArgs(
        namespace=cert_manager_ns_name,
    ),
    opts=pulumi.ResourceOptions(provider=k8s_provider, depends_on=[cert_manager_ns]),
)

Output of pulumi about

CLI          
Version      3.54.0
Go Version   go1.20
Go Compiler  gc

Plugins
NAME                      VERSION
docker                    3.6.1
gcp                       6.46.0
kubernetes                3.23.1
kubernetes-cert-manager   0.0.5
kubernetes-ingress-nginx  0.0.10
python                    unknown

Host     
OS       darwin
Version  13.0.1
Arch     x86_64

This project is written in python: executable='/Users/primet/work/gouach/code/gouach-backend/infra-arch-pulumi/venv/bin/python3' version='3.10.9
'

Current Stack: maelp/infra-arch-pulumi/staging

TYPE                                              URN
pulumi:pulumi:Stack                               urn:pulumi:staging::infra-arch-pulumi::pulumi:pulumi:Stack::infra-arch-pulumi-staging
docker:image:Image                                urn:pulumi:staging::infra-arch-pulumi::docker:image:Image::web-app-image
pulumi:providers:gcp                              urn:pulumi:staging::infra-arch-pulumi::pulumi:providers:gcp::gcp
pulumi:providers:kubernetes                       urn:pulumi:staging::infra-arch-pulumi::pulumi:providers:kubernetes::cluster
pulumi:providers:kubernetes-ingress-nginx         urn:pulumi:staging::infra-arch-pulumi::pulumi:providers:kubernetes-ingress-nginx::default_0_0_10
pulumi:providers:gcp                              urn:pulumi:staging::infra-arch-pulumi::pulumi:providers:gcp::default_6_46_0
kubernetes-ingress-nginx:index:IngressController  urn:pulumi:staging::infra-arch-pulumi::kubernetes-ingress-nginx:index:IngressController::ingress-ctrl
gcp:artifactregistry/repository:Repository        urn:pulumi:staging::infra-arch-pulumi::gcp:artifactregistry/repository:Repository::gcp-docker-repository
pulumi:providers:kubernetes                       urn:pulumi:staging::infra-arch-pulumi::pulumi:providers:kubernetes::default
kubernetes:helm.sh/v3:Release                     urn:pulumi:staging::infra-arch-pulumi::kubernetes-ingress-nginx:index:IngressController$kubernetes:helm.sh/v3:Release::ingress-ctrl-helm
kubernetes:apps/v1:Deployment                     urn:pulumi:staging::infra-arch-pulumi::kubernetes:apps/v1:Deployment::web-app-1
kubernetes:apps/v1:Deployment                     urn:pulumi:staging::infra-arch-pulumi::kubernetes:apps/v1:Deployment::web-app-2
kubernetes:core/v1:Service                        urn:pulumi:staging::infra-arch-pulumi::kubernetes:core/v1:Service::web-app-1-svc
kubernetes:core/v1:Service                        urn:pulumi:staging::infra-arch-pulumi::kubernetes:core/v1:Service::web-app-2-svc
kubernetes:networking.k8s.io/v1:Ingress           urn:pulumi:staging::infra-arch-pulumi::kubernetes:networking.k8s.io/v1:Ingress::web-app-ingress


Found no pending operations associated with staging

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/maelp
User           maelp
Organizations  maelp

Dependencies:
NAME                             VERSION
pip                              23.0.0
pulumi-docker                    3.6.1
pulumi-gcp                       6.46.0
pulumi-kubernetes-cert-manager   0.0.5
pulumi-kubernetes-ingress-nginx  0.0.10
setuptools                       66.0.0
wheel                            0.38.4

Pulumi locates its logs in /var/folders/6d/tgjjs7m13yngmk287mk9cv9h0000gp/T/ by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[maelp]

On the other hand, installing directly from Helm seems to work

cert_manager_chart = Chart(
    "cert-manager",
    config=ChartOpts(
        fetch_opts={"repo": "https://charts.jetstack.io"},
        chart="cert-manager",
        version="v1.11.0",
        namespace=cert_manager_ns_name,
        values={"installCRDs": True},
    ),
    opts=pulumi.ResourceOptions(provider=k8s_provider, depends_on=[cert_manager_ns]),
)

[maelp]

But then, when I add a TLS to my Ingress and deploy with the helm chart, I get this:

  kubernetes:batch/v1:Job (cert-manager/cert-manager-startupapicheck):
    warning: This resource contains Helm hooks that are not currently supported by Pulumi. The resource will be created, but any hooks will not be executed. Hooks support is tracked at https://github.com/pulumi/pulumi-kubernetes/issues/555 -- This warning can be disabled by setting the PULUMI_K8S_SUPPRESS_HELM_HOOK_WARNINGS environment variable
    error: 2 errors occurred:
    	* the Kubernetes API server reported that "cert-manager/cert-manager-startupapicheck" failed to fully initialize or become live: Resource 'cert-manager-startupapicheck' was created but failed to initialize
    	* [BackoffLimitExceeded] Job has reached the specified backoff limit

[maelp]

This might be because I'm deploying to GKE AutoPilot?

[EronWright]

It seems reasonable that the issue is related to AutoPilot, and I updated the title to that effect. Here's a ticket in cert-manager side with some general tips on deploying to AutoPilot:
cert-manager/cert-manager#3717

I would suggest that we re-test on a GKE AutoPilot cluster using the above instructions, and then incorporate the learnings into our documentation.