Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLI-287: add ignore parameter toImage. add_local_dir and Image. copy_local_dir methods #2636

Open
wants to merge 24 commits into
base: main
Choose a base branch
from

Conversation

kramstrom
Copy link
Contributor

@kramstrom kramstrom commented Dec 10, 2024

  • Adds an ignore parameter to our Image add_local_dir and copy_local_dir methods. It is similar to the condition method on Mount methods but instead operates on a Path object. It takes either a list of string patterns to ignore which follows the dockerignore syntax implemented in our PatternMatcher class using a new LocalFileFilter class which wraps around it, or you can pass in a callable which allows for more flexible selection of files.

Usage:

img.add_local_dir(
  "./local-dir", 
  remote_path="/remote-path", 
  ignore=LocalFileFilter("**/*", "!*.txt") # ignore everything except files ending with .txt
)

img.add_local_dir(
  ignore=~LocalFileFilter("**/*.py") # can be inverted for when inclusion filters are simpler to write
)

img.add_local_dir(
  ...,
  ignore=["**/*.py", "!module/*.py"] # ignore all .py files except those in the module directory
)

img.add_local_dir(
  ...,
  ignore=lambda fp: fp.is_relative_to("somewhere") # use a custom callable
)

which will add the ./local-dir directory to the image but ignore all files except .txt files

Copy link
Contributor

@mwaskom mwaskom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a drive-by; saw it's still WIP so apologies if you're thinking about these already.

modal/mount.py Outdated Show resolved Hide resolved
modal/_utils/local_file_filter.py Outdated Show resolved Hide resolved
modal/image.py Outdated Show resolved Hide resolved
@kramstrom kramstrom closed this Dec 11, 2024
@kramstrom kramstrom force-pushed the kramstrom/cli-287-add-include_files-parameter-to-image-dir-copy-methods branch from b7062be to f87f793 Compare December 11, 2024 10:42
…rom/cli-287-add-include_files-parameter-to-image-dir-copy-methods
@kramstrom kramstrom reopened this Dec 11, 2024
@kramstrom kramstrom marked this pull request as ready for review December 12, 2024 15:08
@kramstrom kramstrom requested review from mwaskom and freider December 12, 2024 15:08
modal/mount.py Outdated Show resolved Hide resolved
modal/mount.py Outdated Show resolved Hide resolved
modal/__init__.py Outdated Show resolved Hide resolved
modal/image.py Outdated Show resolved Hide resolved
@@ -23,11 +23,12 @@
from ._utils.async_utils import aclosing, async_map, synchronize_api
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to make any changes to Mount if we're planning to deprecate it as public API? I could see the argument that it would be confusing to have an inversion between the public interface / internal implementation I guess.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think the inversion is annoying but also since the ignore callable works on Path objects and the condition we have for mount methods only allows str you would basically have to wrap every ignore to work on strings instead

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per @freider's comment below I opted to add a private method to the Mount object for the image methods to use instead and leave the signature of the Mount copy & add methods as is

@kramstrom kramstrom force-pushed the kramstrom/cli-287-add-include_files-parameter-to-image-dir-copy-methods branch from a9ebd36 to 697fbf2 Compare December 13, 2024 09:14
modal/_utils/local_file_filter.py Outdated Show resolved Hide resolved
modal/_utils/local_file_filter.py Outdated Show resolved Hide resolved
modal/mount.py Outdated Show resolved Hide resolved
@kramstrom kramstrom changed the title [WIP] CLI-287: add include_files parameter to image dir copy/add methods CLI-287: add include_files parameter to image dir copy/add methods Dec 13, 2024
@kramstrom kramstrom requested review from freider and mwaskom December 13, 2024 10:54
@kramstrom kramstrom changed the title CLI-287: add include_files parameter to image dir copy/add methods CLI-287: add ignore parameter toImage. add_local_dir and Image. copy_local_dir methods Dec 13, 2024
modal/mount.py Outdated Show resolved Hide resolved
from .pattern_matcher import PatternMatcher


class LocalFileFilter:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this should be called "IgnorePatterns" or something, and make it clear in that it excludes on True values. The word "Filter" and some of these docstrings make it sound like its the other way around

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this class in itself doesn't exclude or include specifically though, I guess that's more how it's used in the Image methods, but I'm open to renaming it, but maybe something like FilePathPatternMatcher or something to show that it's related to our PatternMatcher?

Copy link
Contributor

@mwaskom mwaskom Dec 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still a little confused about why we need to be wrapping our own PatternMatcher object here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants