Helm Chart without Parallelism will sometimes deploy the service before the deployment which causes it to hang waiting

[mitchellmaler]

Steps to reproduce

1. deploy helm chart in namespace with parallelism disabled (-p -1)
2. service is deployed before deployment which causes it to hang waiting for pods that wont ever be deployed

Expected: workloads deploy first
Actual: service deployed before workloads

[mikhailshilkov]

@mitchellmaler Could you share more details about your scenario, which chart you are deploying, and why you disable parallelism?

[mitchellmaler]

Hi @mikhailshilkov I was deploying the Rancher chart 2.5.9.
I was disabling parallelism to test that each resource can come up one at a time in the right order. We usually only disable it when we replace multiple nodes that way it only hits one at a time.

[mikhailshilkov]

Thank you for your response. Unfortunately, I expect that chart resources are deployed in a somewhat random order, as there's no obvious way to order them in a general case. How would you define "the right order"? cc @lblackstone

[mitchellmaler]

After some more thought I checked with the helm library ordering and it does place service before deployment which is what I experienced.

https://github.com/helm/helm/blob/release-3.0/pkg/releaseutil/kind_sorter.go#L27

So the main issue is how Pulumi will automatically wait on a service for pods to come up but if you reduce or disable parallelism you might end up stuck.

My thought would be to add some mechanism to logically check that the deployment is still queued up while waiting on a service and kick back that wait until afterwards. This would be in a low or disabled parallelism situation since in the default mode it would wait while the deployment is being or soon will be created.

[lblackstone]

We did sort resources with that ordering in the original TypeScript implementation of our Helm support, but it got dropped at some point since Pulumi automatically retries resource creation. I didn't consider the case where parallelism is explicitly limited, so it may make sense to bring back some ordering. The list that @mitchellmaler linked has Pod, Deployment, etc after Service, so that ordering wouldn't fix this specific problem.

Aside from figuring out an ordering to avoid this problem, you could enable the skipAwait option on the chart. That obviously comes with the downside of not checking resource readiness, but could be an option to get you unstuck for now.

[EronWright]

The upcoming Chart v4 resource has improved support for resource ordering, based on cli-utils as described here. This means that you can apply a config.kubernetes.io/depends-on annotation to ensure that the deployment is installed first.