Idea: vendor external resources including non-kustomize manifests

[yujunz]

Remote build of kustomize could be slow if the repository is huge while we only need a small piece from it. For example, when adding https://github.com/argoproj/argo-cd/tree/master/manifests/cluster-install as a remote resource, kustomize needs to clone the whole repository which takes a long time.

The idea of vendor external resources is similar to go mod vendor which makes vendored copy of dependencies to speed up kustomize build.

If you find it useful too, please 👍 this ticket and I will draft a KEP for further review.

[bzub]

Similar issue: #1132

Vendoring seems like it could be done easily enough outside of kustomization using git submodule, git subtree or simple forking/cloning.

To me it would only be worth mimicking go modules if kustomize also supported similar go.mod/go.sum validations. Otherwise the same issues with latency and security for remote bases would remain that exist today -- a remote not under your control could change at any time.

Here's another proposal that could help with speed via caching, as a compromise: #1431

[bzub]

The ability to specify a git clone depth like go-getter does could satisfy a lot of use-cases in a simple way.

[yujunz]

We did implement a go-getter plugin as in https://github.com/kubernetes-sigs/kustomize/tree/master/plugin/someteam.example.com/v1/gogetter .

[yujunz]

go-getter supports checksum validation, subdirectory and almost everything we want for dependency management. The current drawbacks is it requires extra configuration steps to enable the plugin and building with alpha flag. Maybe we just need to connecting the dots to make life easier for end users.

[yujunz]

A quick example in https://github.com/yujunz/kustomize-vendor-example

[pietervincken]

If I'm not mistaken the git clone is already performing a shallow clone? (see

kustomize/api/git/cloner.go

Line 69 in 644f2dd

"--depth=1",

). Are you sure it is slow due to the git clone?

We've experienced slow (30-60s) builds as well, but from the verbose logging is seems that it's actually all the processing that is performed, not the git clone. The CPU (single core) is also pinned at 100% during this time.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[bb010g]

I just ran into this when packaging the Flux CLI for Nixpkgs due to it running kustomize build on manifests specified as HTTPS URLs. A simple kustomize vendor command would be great for package maintainers.