-
Notifications
You must be signed in to change notification settings - Fork 169
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Address shadowing divergence in reftest, update semantics doc #1201
base: main
Are you sure you want to change the base?
Conversation
b3d7045
to
f405d75
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is the right change and semantics, I've put some feedback on docs mainly
doc/SEMANTICS.md
Outdated
For concurrently writing to the same object from multiple instances, we do not place any guarantees on which version of the object will be written to S3. | ||
Thus, if two clients, at least one of them using Mountpoint, write to the same file and there is some overlap, Mountpoint will never read partial or corrupt data -- however there are no guarantees, which version ultimately is written to S3. | ||
This is mainly determined by the S3 semantics and the specific kinds of requests the clients use to write to S3. | ||
If you use multiple instances for writing the same key (or another client modifies the Object while the file is open for writing), be aware that Mountpoint currently uses Multipart Uploads (MPU) that are created after the file is opened to upload the data to S3. | ||
This implies, that the provisions in the [S3 Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html#distributedmpupload) regarding multiple concurrent MPUs apply. | ||
For example, if (in a versioning-enabled bucket) you open a file for writing on the first MP instance at 11 AM, and if you open the same file on another instance at 12 AM, and finish writing on the first instance after the second, | ||
the version of the second instance will be committed to S3, and the 11 AM version will not be accessible from any of the Mountpoint instances afterwards. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would cut this to be honest. We already recommend above not to write to the same object concurrently, and this is far too much detail for this half of the semantics doc. (We split it into an overview for most readers, then a detailed section below that goes into specific S3 and file system behaviors.)
If needed, we could mention in the detailed semantics later in the doc but delegate reading to the linked documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html#distributedmpupload.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I cut this and added one sentence to the end of the file -- but I'm not too sure if we should keep that or just cut it completely.
doc/SEMANTICS.md
Outdated
then mounting your bucket would give a file system with a `blue` directory, containing the file `image.jpg`. The `blue` object will not be accessible. Deleting the key `blue/image.jpg` will remove the `blue` directory, and cause the `blue` file to become visible. | ||
Additionally, remote content shadows local content. Thus Mountpoint shadows in the order: remote directories > any local state > remote files. | ||
Thus, for example, if a user creates a local directory, and then a conflicting | ||
object appears in the remote bucket, the directory will still be | ||
visible instead of the new file. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
then mounting your bucket would give a file system with a `blue` directory, containing the file `image.jpg`. The `blue` object will not be accessible. Deleting the key `blue/image.jpg` will remove the `blue` directory, and cause the `blue` file to become visible. | |
Additionally, remote content shadows local content. Thus Mountpoint shadows in the order: remote directories > any local state > remote files. | |
Thus, for example, if a user creates a local directory, and then a conflicting | |
object appears in the remote bucket, the directory will still be | |
visible instead of the new file. | |
then mounting your bucket would give a file system with a `blue` directory, containing the file `image.jpg`. The `blue` object will not be accessible. Deleting the key `blue/image.jpg` will remove the `blue` directory, and cause the `blue` file to become visible. | |
Additionally, remote directories will always shadow local directories and files. Thus, Mountpoint shadows directory entries in the following order where the first takes precedence: remote directories, any local state, and finally remote files. | |
For example, if a user creates a local directory, and then a conflicting | |
object appears remotely in the S3 bucket, the directory will still be | |
visible instead of the new file. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, I mainly incorporated this -- I tried to additionally keep the example similar (using colours) / not switch from second to third person.
/* | ||
Ensure that local files are shadowed by the remote directories. | ||
*/ | ||
#[test] | ||
fn regression_put_nested_over_open_file() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/* | |
Ensure that local files are shadowed by the remote directories. | |
*/ | |
#[test] | |
fn regression_put_nested_over_open_file() { | |
/* | |
Ensure that local files are shadowed by the remote directories. | |
*/ | |
#[test] | |
fn regression_put_shadowing_new_local_file() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
// If a directory of this name exists, ignore any local file | ||
if let Some(node) = children.get(dir) { | ||
if node.node_type() == NodeType::Directory { | ||
return true; | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should return false
, no? The node was not added.
// If a directory of this name exists, ignore any local file | |
if let Some(node) = children.get(dir) { | |
if node.node_type() == NodeType::Directory { | |
return true; | |
} | |
} | |
// If a directory of this name exists, ignore any local file | |
if let Some(node) = children.get(dir) { | |
if node.node_type() == NodeType::Directory { | |
return false; | |
} | |
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, thanks for catching that. I'll also update the comment for the method to reflect the implemented behaviour.
This PR closes #1066. |
This commit addresses a case where MP model and property tests diverge (awslabs#1066). The issue was caused by the reference not correctly implementing the shadowing order defined in [#4f8cf0b](awslabs@4f8cf0b). Additionally, it clarifies the Semantics arising from concurrent MPUs. Signed-off-by: Christian Hagemeier <[email protected]>
f405d75
to
c5f1469
Compare
This commit addresses a case where MP model and property tests diverge (#1066). The issue was caused by the reference not correctly implementing the shadowing order defined in #4f8cf0b. This commit fixes the reference model, and clarifies the semantics arising from concurrent MPUs.
This is not a breaking change, as it only impacts the reference tests.
This does not need a Changelog entry, as the change does not impact Mountpoint's behaviour.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).