Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support hashicorp/go-getter format http url as remote base #923

Closed
3 tasks
yujunz opened this issue Mar 27, 2019 · 18 comments · Fixed by #2169
Closed
3 tasks

Support hashicorp/go-getter format http url as remote base #923

yujunz opened this issue Mar 27, 2019 · 18 comments · Fixed by #2169
Labels
kind/feature Categorizes issue or PR as related to a new feature.

Comments

@yujunz
Copy link
Member

yujunz commented Mar 27, 2019

Original requirement from #894 .

Currently only local files and git is supported in kustomzie. When the bases is stored in a large mono-repo, let's say giga bytes, it is not feasible to clone the whole repository to fetch just one base.

The following features in hashicorp/go-getter satisfies such requirements perfectly, i.e.

@Liujingfang1
Copy link
Contributor

Please note that we don't plan to use hashicorp/go-getter in Kustomize since it brings a large number of dependencies.
It would be nice to have Kustomize work flexible with different types of remote targets, but the implementation should be
either inside Kustomize or dependent on a very small number of libraries.

@yujunz
Copy link
Member Author

yujunz commented Mar 28, 2019

we don't plan to use hashicorp/go-getter

I think I saw it in code somewhere. Is it already removed? It does not have to be hashicorp/go-getter. Just need support for go-getter format http url.

@yujunz yujunz changed the title Support hashicorp/go-getter for https tarball source Support hashicorp/go-getter format http url as remote base Mar 28, 2019
@Liujingfang1 Liujingfang1 added the kind/feature Categorizes issue or PR as related to a new feature. label Mar 28, 2019
@objectiveous
Copy link

+1.

@pohly
Copy link
Contributor

pohly commented Apr 12, 2019

I'm a bit confused. The documentation says that bases entries can have hashicorp URL format:

# The entry could be a relative path pointing to a local directory
# or a url pointing to a directory in a remote repo.
# The url should follow hashicorp/go-getter URL format
# https://github.com/hashicorp/go-getter#url-format

Is that incorrect and the actual implementation expects something else, or only supports a subset of the hashicorp implementation?

@yujunz
Copy link
Member Author

yujunz commented Apr 19, 2019

I saw some recent update about plugin (see f6e01cf )

Is it feasible to exec go-getter as a plugin? It has a standalone executable. (see https://github.com/hashicorp/go-getter#installation-and-usage).

@Liujingfang1

@Liujingfang1
Copy link
Contributor

@yujunz You could write a generator plugin which executes go-getter. This plugin can fetch the files or directories with only raw K8s resources.

The generator plugins assumes the output is a list of K8s resources. If the remote URL doesn't have a kustomization.yaml and only contains raw K8s resources, the plugin can just load them and write them into a resmap.

If the remote URL contains a kustomization.yaml, the plugin has to run a kustomize build and then write the output into a resmap. In the case, the plugin depends on kustomize itself, which seem like a improper direction.

@yujunz
Copy link
Member Author

yujunz commented Apr 20, 2019

Is plugin limited to generator?

Can we also extend loader with external plugin?

@Liujingfang1
Copy link
Contributor

Currently plugins are either generators or transformers, loaders are not included. If you have any thoughts, feel free to write a KEP for it.

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 21, 2019
@yujunz
Copy link
Member Author

yujunz commented Aug 13, 2019

A simple plugin for go-getter: https://github.com/kustless/kustomize-plugins/tree/master/gogetter

@yujunz
Copy link
Member Author

yujunz commented Aug 14, 2019

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 14, 2019
@JohnPaton
Copy link

So just to be clear, the comment that @pohly referenced, and the remote build example, are not accurate currently?

@yujunz
Copy link
Member Author

yujunz commented Aug 28, 2019

What the examples says is true.

The url should follow hashicorp/go-getter URL format. Here are some example urls pointing to Github repos following this convention.

What is also true is that only a subset of all hashicorp/go-getter URL formats are supported.

@JohnPaton
Copy link

JohnPaton commented Aug 29, 2019

Thanks for clarifying.

For other people who are searching: I was asking this because I was trying to use a remote resource or base from a private repository over ssh. The correct format turned out to be:

bases:
  - ssh://github.com/username/repo_name.git//path/to/dir?ref=branch_name

@pohly
Copy link
Contributor

pohly commented Aug 29, 2019

What is also true is that only a subset of all hashicorp/go-getter URL formats are supported.

And which subset is that?

@yujunz
Copy link
Member Author

yujunz commented Aug 29, 2019

@Liujingfang1
Copy link
Contributor

@yujunz Would you like to add this generator plugin https://github.com/kustless/kustomize-plugins/tree/master/gogetter as an example generator in Kustomize repo and write some docs for it?

yujunz added a commit to yujunz/kustomize that referenced this issue Aug 30, 2019
yujunz added a commit to yujunz/kustomize that referenced this issue Aug 30, 2019
@yujunz
Copy link
Member Author

yujunz commented Sep 6, 2019

@yujunz yujunz closed this as completed Sep 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants