-
Notifications
You must be signed in to change notification settings - Fork 178
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
document that directories are flattened unless data_path, strip_prefix are set to "." #82
Comments
This look like a bug to me. Normally default should be "." so not strip the path. |
Yeah, this same flattening-every-dir problem seems to be affecting pkg_tar. I've just added some targets that demonstrate it in pkg_tar, too. I also added another file at Changing pkg_tar to use There might be a middle ground we need to hit where the default doesn't flatten all of the directories, but also doesn't include the dirs that the filegroup was made by globbing over. That'd get us closer to what people might expect, but not being able to rewrite filegroups target directories easily when creating a pkg_tar or docker_build from multiple targets would still be a bummer. Lmk what you think. |
This sounds reasonable to me. Can you ask the question on bazel-discuss@ to see what other thinks? Wrapping it with a macro for people might be an acceptable workaround for people that complains. |
Nobody bit on that thread! Seems like it's up to y'all. |
Ok thanks, this will get done next quarter. Happy holidays! |
I know I'm late to the discussion here. However, I just ran into this problem, so I thought I'd cast a vote in the event that it's not too late. Personally, I'd prefer to see the default behavior be to preserve the directory structure below the BUILD file containing the filegroup rule, then allow options like |
@damienmg FWIW +1 to making the default preserve directory structure. I just got bit by this playing with some samples and it probably would have burned way more time if I didn't have a feint recollection of this issue from triaging |
Just got hit by this again. Any updates? |
docker_build should be fixed IIUC |
Still happening at container_image |
I'm no longer seeing this behavior in (In fact, using my |
Oh, the So now we're in a really weird spot where you have to split up your |
IIUC that change was something that bit Having been bitten by this strange default behavior multiple times myself, I'm completely on-board that this behavior is bizarre and not what users expect. :) We should probably add a flag to restore the behavior as it exists today, in case folks are broken by it. It would be useful to have a Googler to shepherd this into the mono-repo (I don't have time right now). |
Having to add |
These rules, If we add the
It also ignores the We even tried setting the |
In case anyone else runs into this, I just wasted a few hrs figuring out that: The strip_prefix="." only works if the pkg_tar target is in the top level BUILD file. If it is down a couple levels it does not work and you get a flattened tree. You can see that the helper function _short_path_dirname special cases top level targets |
From docs in rules_docker (https://github.com/bazelbuild/rules_docker#container_image) Hint: if you want to put files in specific directories inside the image use @shettich , can you try with |
I was not building a container image, just a tar. You can see in the code that the prefix to strip off is calculated by compute_data_path() based on the output path and the strip_prefix your provide. If you provide strip_prefix="." it will use _short_path_dirname() on the output file as the prefix. That function special cases top level target paths, which is why the strip_prefix trick works for some targets but not all. |
I agree with @cgoy-nvidia that pkg_tar is pretty much broken. We need a new design for how fileset's and/or other targets are combined into a tar/zip/deb/rpm, ... file. Moving this to rules_pkg so we can work on it there. |
... it's still pretty weird in 1.0; might a PR be accepted to note this on the pkg_tar doc page, between now and when pkg_tar gets revamped? |
As the original reporter who just got bit by this again today, it would be lovely to see a fix. What direction do y'all want to take it in? |
Oh, wonderful! Thank you so much for the update! |
Related to #354 |
This is a fix for bazelbuild/rules_pkg#82 Change-Id: I7733ea110103298c36106e40233d57fa9b1d2a1a
This is a fix for bazelbuild/rules_pkg#82 Change-Id: I7733ea110103298c36106e40233d57fa9b1d2a1a Reviewed-on: https://review.monogon.dev/c/NetMeta/+/1115 Reviewed-by: Leopold Schabel <[email protected]>
In bazel 0.4.1 (and I think 0.3.2), if you give docker_build a filegroup with a glob that contains multiple directories of files, those files will be flattened into one directory.
Here's the repo I made for the reproduction. It needed a bit of set up. (You'll need
xz
installed to get the wheezy rootfs file unpacked.)If you run
bazel run //subdir:buggy_flatten_image
, I expected a/foo/quux
and a/bar/quux
file to exist in the created image but instead only a/quux
exists (verifiable with adocker run -it bazel/subdir:buggy_flatten_image
). A warning about duplicate filepaths is given but not error occurs.The addition of
data_path = "."
as in the//subdir:has_subdirs_image
target fixes this and that's very surprising.I think it's inheriting some behavior from pkg_tar's strip_prefix maybe, but it's definitely not what I expected and finding the solution wasn't easy, either. This also makes it difficult to merge together multiple filegroups with subdirectories into one image because you have to give each of them their own docker_build (or, perhaps, pkg_tar) in order to keep them from ending up at
/
.Seems like this, or something similar, was addressed in bazelbuild/bazel#677 / bazelbuild/bazel@0e22258 but maybe something has happened in between then and now.
(I noticed this when a build was throwing up warnings, but not errors, of duplicate filepaths being included. Maybe that should be another ticket?)
Environment info
macOS 10.12 Sierra
bazel info release
):release 0.4.1-homebrew
The text was updated successfully, but these errors were encountered: