Support hashicorp/go-getter format http url as remote base

[yujunz]

Original requirement from #894 .

Currently only local files and git is supported in kustomzie. When the bases is stored in a large mono-repo, let's say giga bytes, it is not feasible to clone the whole repository to fetch just one base.

The following features in hashicorp/go-getter satisfies such requirements perfectly, i.e.

• subdirectory
• checksumming
• unarchiving

[Liujingfang1]

Please note that we don't plan to use hashicorp/go-getter in Kustomize since it brings a large number of dependencies.
It would be nice to have Kustomize work flexible with different types of remote targets, but the implementation should be
either inside Kustomize or dependent on a very small number of libraries.

[yujunz]

we don't plan to use hashicorp/go-getter

I think I saw it in code somewhere. Is it already removed? It does not have to be hashicorp/go-getter. Just need support for go-getter format http url.

[objectiveous]

+1.

[pohly]

I'm a bit confused. The documentation says that bases entries can have hashicorp URL format:

kustomize/docs/kustomization.yaml

Lines 130 to 133 in 177297c

	# The entry could be a relative path pointing to a local directory
	# or a url pointing to a directory in a remote repo.
	# The url should follow hashicorp/go-getter URL format
	# https://github.com/hashicorp/go-getter#url-format

Is that incorrect and the actual implementation expects something else, or only supports a subset of the hashicorp implementation?

[yujunz]

I saw some recent update about plugin (see f6e01cf )

Is it feasible to exec go-getter as a plugin? It has a standalone executable. (see https://github.com/hashicorp/go-getter#installation-and-usage).

@Liujingfang1

[Liujingfang1]

@yujunz You could write a generator plugin which executes go-getter. This plugin can fetch the files or directories with only raw K8s resources.

The generator plugins assumes the output is a list of K8s resources. If the remote URL doesn't have a kustomization.yaml and only contains raw K8s resources, the plugin can just load them and write them into a resmap.

If the remote URL contains a kustomization.yaml, the plugin has to run a kustomize build and then write the output into a resmap. In the case, the plugin depends on kustomize itself, which seem like a improper direction.

[yujunz]

Is plugin limited to generator?

Can we also extend loader with external plugin?

[Liujingfang1]

Currently plugins are either generators or transformers, loaders are not included. If you have any thoughts, feel free to write a KEP for it.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[yujunz]

A simple plugin for go-getter: https://github.com/kustless/kustomize-plugins/tree/master/gogetter

[yujunz]

/remove-lifecycle stale

[JohnPaton]

So just to be clear, the comment that @pohly referenced, and the remote build example, are not accurate currently?

[yujunz]

What the examples says is true.

The url should follow hashicorp/go-getter URL format. Here are some example urls pointing to Github repos following this convention.

What is also true is that only a subset of all hashicorp/go-getter URL formats are supported.

[JohnPaton]

Thanks for clarifying.

For other people who are searching: I was asking this because I was trying to use a remote resource or base from a private repository over ssh. The correct format turned out to be:

bases:
  - ssh://github.com/username/repo_name.git//path/to/dir?ref=branch_name

[pohly]

What is also true is that only a subset of all hashicorp/go-getter URL formats are supported.

And which subset is that?

[yujunz]

Only some of the git repo spec

See https://github.com/kubernetes-sigs/kustomize/blob/master/pkg/git/repospec_test.go

[Liujingfang1]

@yujunz Would you like to add this generator plugin https://github.com/kustless/kustomize-plugins/tree/master/gogetter as an example generator in Kustomize repo and write some docs for it?

[yujunz]

Please refer to https://github.com/kubernetes-sigs/kustomize/blob/master/examples/goGetterGeneratorPlugin.md