[feature] cache helm YAML rendered result

[brandonkal]

Problem description

Helm chart index lookup, fetch, template, and rendering consumes a notable amount of time. This is especially painful when a stack has many helm charts and unrelated changes are made to the stack. I have a slow internet connection which makes this even worse.

I'd also like to be able to distribute Pulumi projects without requiring the user to install the helm 3 binary on their systems. If the rendered YAML string can be cached and stored in git, this would enable that use case as well.

Affected product version(s)

All

Suggestions for a fix

The work performed in the parseTemplate method up to and including this line:

pulumi-kubernetes/pkg/gen/nodejs-templates/helm/v2/helm.ts

Line 243 in 127ed57

yaml: objs.map(o => jsyaml.safeDump(o)),

can be cached to disk. This only requires improvements to the SDK. Caching can be done in cases where the inputs are known and unchanged and a chart version is specified. It can also be done across stacks, as long as the chart inputs remain the same. Pulumi can create a .pulumi-cache/helm-urn.yaml file that is used whenever possible to conserve work.

[brandonkal]

Another thought: If the helm chart happens to generate things that pulumi considers to be secrets, pulumi may wish to ensure they are encrypted before writing anything to disk. Or perhaps just using a single key to encrypt the whole file.

[Place1]

Maybe a stepping stone could be to cache the helm chart's .tgz locally. That would at least allow repeat runs to be sped up without Pulumi needing to solve the "save a secret" problem highlighted above.

Some of my stacks end up downloading 10+ charts per run making pulumi very sluggish.

[Place1]

I actually just ran a test. I implemented a little local cache for the chart's i'm using. My preview time went from 103 seconds to 37 seconds.

My test project uses 14 helm charts

I cached the charts by downloading/untar'ing them to a directory and then using Pulumi Helm's path option to use the local chart.

[Bazze]

This would be a well needed addition. Overtime we tend to use a lot of helm charts and our Pulumi programs are just getting slower and slower 😅 I'm just waiting for the day when one of the Helm charts won't longer be public available on the internet – then a copy of the helm chart under version control would be a great saviour.

[viveklak]

As mentioned in #1504, this may also help in case of network issues.

[raelix]

Would be better at least if we could have the ability of downloading the charts in parallel. Is there any way to achieve this? I have to install more than 100 charts so this becomes really slow...

[EronWright]

There's a lot of good suggestions in the above discussion, here's a recap:

1. Parallelism - in some SDKs, the charts are downloaded sequentially. I believe this will be addressed with Chart v4 because MLCs are constructed in parallel.
2. Helm binary - fortunately there's no need to have the helm tool because Pulumi now uses Helm as a library.
3. Chart caching - see: HIP for download cache (charts & provenance files)  helm/community#185