-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Go SDK]: Enable fileio.MatchContinuously to emit duplicate file if modified #26524
[Go SDK]: Enable fileio.MatchContinuously to emit duplicate file if modified #26524
Conversation
R: @lostluck |
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
Thanks for your patience. I was on vacation towards the end of last week, and just caught up on things. Looking at this now. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thank you.
Sorry for the delay. Some critical distractions occurred after my initial look.
} | ||
} | ||
|
||
return metadata, nil | ||
} | ||
|
||
func lastModified(ctx context.Context, fs filesystem.Interface, path string) (time.Time, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not in this PR, but we could consider putting this as a helper function in the filesystem package instead, to allow more reasonable re-use and consistency in the implementation:
Eg. We implement copy, but if there's a Copier implementation, it can be used instead.
https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/io/filesystem/util.go#L55
We'd probably want a "creation time" too though when we do that, so the default isn't the zero time. Not critical at this stage though.
Oh dang, I will send a quick fix for this, but it looks like the time range might be incorrectly set for the local unit test: Caught in another CL.
|
Updates the
fileio.MatchContinuously
transform to support emission of duplicate matches if a file has been modified since it was last observed. The behavior is enabled by passing thefileio.MatchDuplicateAllowIfModified()
option. Fixes #26523.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123
), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>
instead.CHANGES.md
with noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.