From aac7fc98c460272b113038333f0091152afbdc6d Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 13 Jun 2022 13:36:51 -0600 Subject: [PATCH 01/49] prelim v2 sketch --- file-transfer-v2-complex.seqdiag | 32 +++++ file-transfer-v2-simple.seqdiag | 15 +++ file-transfer-v2.md | 206 +++++++++++++++++++++++++++++++ 3 files changed, 253 insertions(+) create mode 100644 file-transfer-v2-complex.seqdiag create mode 100644 file-transfer-v2-simple.seqdiag create mode 100644 file-transfer-v2.md diff --git a/file-transfer-v2-complex.seqdiag b/file-transfer-v2-complex.seqdiag new file mode 100644 index 0000000..1155149 --- /dev/null +++ b/file-transfer-v2-complex.seqdiag @@ -0,0 +1,32 @@ +seqdiag { + Alice -> Bian [label="OPEN(subchannel=1)"] + Alice -> Bian [label="control \n Offer[filename=foo, subchannel=1, id=42]"] + Alice -> Bian [label="OPEN(subchannel=3)"] + Alice -> Bian [label="control \n Offer[filename=bar, subchannel=3, id=89]"] + + Alice <- Bian [label="control \n Accept[id=42]"] + Alice -> Bian [label="subchannel 1 \n DATA"] + + Alice -> Bian [label="OPEN(subchannel=2)"] + Alice <- Bian [label="control \n Offer[filename=quux, subchannel=2, id=65]"] + + Alice <- Bian [label="control \n Accept[id=89]"] + Alice -> Bian [label="subchannel 3 \n DATA"] + Alice -> Bian [label="subchannel 1 \n DATA"] + Alice -> Bian [label="subchannel 3 \n DATA"] + Alice -> Bian [label="subchannel 1 \n DATA"] + + Alice -> Bian [label="control \n Accept[id=65]"] + Alice <- Bian [label="subchannel 2 \n DATA"] + + Alice -> Bian [label="CLOSE(subchannel=1)"] + + Alice <- Bian [label="subchannel 2 \n DATA"] + Alice -> Bian [label="subchannel 3 \n DATA"] + Alice -> Bian [label="subchannel 3 \n DATA"] + + Alice <- Bian [label="CLOSE(subchannel=2)"] + Alice -> Bian [label="CLOSE(subchannel=3)"] + + Alice -> Bian [label="close mailbox"] +} diff --git a/file-transfer-v2-simple.seqdiag b/file-transfer-v2-simple.seqdiag new file mode 100644 index 0000000..8a50b67 --- /dev/null +++ b/file-transfer-v2-simple.seqdiag @@ -0,0 +1,15 @@ +seqdiag { + Alice -> Bian [label="OPEN(subchannel=1)"] + Alice -> Bian [label="control \n Offer[filename=foo, subchannel=1, id=42]"] + + Alice <- Bian [label="control \n Accept[id=42]"] + + Alice -> Bian [label="subchannel 1 \n DATA"] + Alice -> Bian [label="subchannel 1 \n DATA"] + Alice -> Bian [label="subchannel 1 \n DATA"] + + Alice -> Bian [label="CLOSE(subchannel=1)"] + + Alice <- Bian [label="close mailbox"] + Alice -> Bian [label="close mailbox"] +} diff --git a/file-transfer-v2.md b/file-transfer-v2.md new file mode 100644 index 0000000..50bf3bd --- /dev/null +++ b/file-transfer-v2.md @@ -0,0 +1,206 @@ +# File-Transfer Protocol v2 + +This version 2 protocol is a complete replacement for the original (referred to as "v1") file-transfer protocol. + +Both sides must support and use Dilation (see `dilation-protocol.md`). + + +## Overview and Features + +We describe a flexible, "session"-based approach to file transfer allowing either side to offer files to send (while the other side may accept or reject each offer). +Either side may terminate the transfer session. + +File are sent individually, with no dependency on zip or other archive formats. + +Metadata is included in the offers to allow the receiver to decide if they want that file before the transfer begins. + + +## Protocol Details + +See the Dilation document for details on the Dilation setup procedure. +Once a Dilation-supporting connection is open, we will have a "control" subchannel (subchannel #0). + +All offers are sent over the control channel. +All answers (accept or reject) are also sent over the control channel. +All control-channel messages are encoded using `msgpack`. + --> XXX: see "message encoding" discussion + +Control-channel message formats are described using Python3 pseudo-code to illustrate the exact data types involved. + +All control-channel messages contain an integer "kind" field describing the type of message. + +Offer messages look like this: + +```python +class Offer: + kind: int = 1 # "offer" + id: int # unique random identifier for this offer + filename: str # utf8-encoded unicode relative pathname + timestamp: int # Unix timestamp (seconds since the epoch in GMT) + bytes: int # total number of bytes in the file + subchannel: int # the subchannel which the file will be sent on +``` + +The subchannel in an Offer MUST NOT match any subchannel in any existing Offer from this side nor from the other side. +This is enforced by the Dilation implementation: the Leader allocates only odd channels (starting with 1) and the Follower allocates only even channels (starting with 2). +That is, the side producing the Offer first opens a subchannel and then puts the resulting ID into the Offer message. + +There are two kinds of repies to an offer, either an Accept message or a Reject message. +Reject messages look like this: + +```python +class OfferReject: + kind: int = 2 # "offer reject" + id: int # matching identifier for an existing offer from the other side + reason: str # utf8-encoded unicode string describing why the offer is rejected +``` + +Accept messages look like this: + +```python +class OfferAccept: + kind: int = 3 # "offer accpet" + id: int # matching identifier for an existing offer from the other side +``` + +When the offering side gets an `OfferReject` message, the subchannel is immediately closed. +The offering side may show the "reason" string to the user. +This offer ID should never be re-used during this session. + +When the offering side gets an `OfferAccept` message it begins streaming the file over the already-opened subchannel. +When completed, the subchannel is closed. + +That is, the offering side always initiates the open and close of the corresponding subchannel. + +See examples down below. + + +## Discussion and Open Questions + +* (sub)versioning of this protocol + +It is likely useful to assign version numbers to either the protocol or to Offer messages. +This would allow future extensions such as adding more metadata to the Offer (e.g. `thumbnail: bytes`). + +The simplest would be to version the entire protocol. +If we versioned the Offer messages, it's not clear what should happen if an old client receives a too-new Offer. Presumably, reject it -- but it would be better to let the sender know that the client is old and to re-send an older style offer. + +To version the entire protocol we'd have to have each side send an initial control-channel message indicating what version they support. +Perhaps this could be stated as "features" instead (e.g. "thumbnail" feature if we added that, or "extended metadata" if we add more metadata, etc). +Using "features" would have the added semantic benefit that two up-to-date clients may still not want (or be able to use) particular features (e.g. a CLI client might not display thumbnails) but is more complex for both sides. + +Preliminary conclusion: a simple Version message is sent first, with version=0. + +* message encoding + +While `msgpack` is mentioned above, there are several other binary-supporting libraries worth considering. +These are (in no particular order) at least: CBOR or flatbuffers or protocolbuffers or cap'n'proto + + +## Example 1 + +Alice contacts Bian to transfer a single file. + +* The software on Alice's computer begins a Dilation-enabled session, producing a secret code. +* Alice sends this code to Bian +* Software on Bian's computer uses the code to complete the Dilation-enabled session. + +At this point, Alice and Bian are connected (possibly directly, possibly via the relay). +Alice is the "Leader" in the Dilation protocol. +From this point on, the "file transfer v2" protocol is spoken (see below for `seqdiag` markup to render a diagarm). + +* Alice opens a new subchannel (id=1, because she's the Leader) +* Alice sends an Offer to Bian on the control channel +* Bian accepts the Offer +* Alice sends all data on subchannel 1 +* Alice closes subchannel 1 +* Bian closes the mailbox +* Alice also closes the mailbox (which is now de-allocated on the server) + +Here is a sequence diagram of the above. + +```seqdiag +seqdiag { + Alice -> Bian [label="OPEN(subchannel=1)"] + Alice -> Bian [label="control \n Offer[filename=foo, subchannel=1, id=42]"] + + Alice <- Bian [label="control \n Accept[id=42]"] + + Alice -> Bian [label="subchannel 1 \n DATA"] + Alice -> Bian [label="subchannel 1 \n DATA"] + Alice -> Bian [label="subchannel 1 \n DATA"] + + Alice -> Bian [label="CLOSE(subchannel=1)"] + + Alice -> Bian [label="close mailbox"] +} +``` + +## Example 2 + +Alice contacts Bian to start a file-transfer session, sending 2 files and receiving 1. + +The software is started on Alice's computer which initiates a (Dilation-enabled) connection. +Alice communcates the secret code to Bian. +On Bian's computer, the (Dilation-enabled) software completes the connection. + +So, we have a Dilation-enabled connection between Alice and Bian's computers. + +```seqdiag +seqdiag { + Alice -> Bian [label="OPEN(subchannel=1)"] + Alice -> Bian [label="control \n Offer[filename=foo, subchannel=1, id=42]"] + Alice -> Bian [label="OPEN(subchannel=3)"] + Alice -> Bian [label="control \n Offer[filename=bar, subchannel=3, id=89]"] + + Alice <- Bian [label="control \n Accept[id=42]"] + Alice -> Bian [label="subchannel 1 \n DATA"] + + Alice <- Bian [label="OPEN(subchannel=2)"] + Alice <- Bian [label="control \n Offer[filename=quux, subchannel=2, id=65]"] + + Alice <- Bian [label="control \n Accept[id=89]"] + Alice -> Bian [label="subchannel 3 \n DATA"] + Alice -> Bian [label="subchannel 1 \n DATA"] + Alice -> Bian [label="subchannel 3 \n DATA"] + Alice -> Bian [label="subchannel 1 \n DATA"] + + Alice -> Bian [label="control \n Accept[id=65]"] + Alice <- Bian [label="subchannel 2 \n DATA"] + + Alice -> Bian [label="CLOSE(subchannel=1)"] + + Alice <- Bian [label="subchannel 2 \n DATA"] + Alice -> Bian [label="subchannel 3 \n DATA"] + Alice -> Bian [label="subchannel 3 \n DATA"] + + Alice <- Bian [label="CLOSE(subchannel=2)"] + Alice -> Bian [label="CLOSE(subchannel=3)"] + + Alice -> Bian [label="close mailbox"] + Alice <- Bian [label="close mailbox"] +} +``` + +Breaking down all those messages at a high level this is what's happening: + +* Alice is the "Leader" in the Dilation session (this affects the subchannel IDs). +* Alice opens subchannel 1 to allocate an id to send in the first Offer +* Alice sends the first Offer, of file `foo`. +* Alice opens subchannel 3 for the second Offer +* Alice sends the second Offer, of file `bar` +* Bian accepts the Offer for `foo`. +* Alice starts sending data for `foo` on subchannel 1. +* Bian opens subchannel 2 +* Bian sends Offer of file `quux` +* Bian accepts Offer 89, for file `bar` +* Alice sends more data (2 packets each of `foo` and `bar` files) +* Alice accepts Offer 65, for file `quux` +* Bian sends first data of file `quux` +* Alice is done sending file `foo` and closes the subchannel +* Bian sends some data for `quux` +* Alice sends 2 more chunks of data for `bar` +* Bian is done sending `quux` and closes the subchannel +* Alice is done sending `bar` and closes the subchannel +* Alice indicates she is done with the session and closes the mailbox +* Bian acknowledges and also closes the mailbox From 1f454ee4ba0e70358165a2145a910a59182e037e Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 13 Jun 2022 13:46:05 -0600 Subject: [PATCH 02/49] clarify --- file-transfer-v2.md | 34 ++++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/file-transfer-v2.md b/file-transfer-v2.md index 50bf3bd..afa41fa 100644 --- a/file-transfer-v2.md +++ b/file-transfer-v2.md @@ -4,11 +4,13 @@ This version 2 protocol is a complete replacement for the original (referred to Both sides must support and use Dilation (see `dilation-protocol.md`). +Any all-caps words ("MAY", "MUST", etc) follow RFC2119 conventions. + ## Overview and Features We describe a flexible, "session"-based approach to file transfer allowing either side to offer files to send (while the other side may accept or reject each offer). -Either side may terminate the transfer session. +Either side MAY terminate the transfer session. File are sent individually, with no dependency on zip or other archive formats. @@ -20,15 +22,26 @@ Metadata is included in the offers to allow the receiver to decide if they want See the Dilation document for details on the Dilation setup procedure. Once a Dilation-supporting connection is open, we will have a "control" subchannel (subchannel #0). -All offers are sent over the control channel. -All answers (accept or reject) are also sent over the control channel. -All control-channel messages are encoded using `msgpack`. +All offers MUST be sent over the control channel. +All answers (accept or reject) MUST also sent over the control channel. +All control-channel messages MUST be encoded using `msgpack`. --> XXX: see "message encoding" discussion Control-channel message formats are described using Python3 pseudo-code to illustrate the exact data types involved. All control-channel messages contain an integer "kind" field describing the type of message. +Both sides MUST immediately send a Version message on the control channel (XXX see "(sub)versioning of this protocol" dicussion). +Version messages look like this: + +```python +class Version: + kind: int = 0 # "version" + version: int = 1 # the only existing (sub)version of "file transfer v2" +``` + +Either side MAY send any number of Offer messages at any time. +They MUST first open a subchannel to receive the subchannel ID. Offer messages look like this: ```python @@ -41,11 +54,12 @@ class Offer: subchannel: int # the subchannel which the file will be sent on ``` +The `id` in an Offer MUST NOT match any other Offer from this side. The subchannel in an Offer MUST NOT match any subchannel in any existing Offer from this side nor from the other side. -This is enforced by the Dilation implementation: the Leader allocates only odd channels (starting with 1) and the Follower allocates only even channels (starting with 2). +This latter constraint is enforced by the Dilation implementation: the Leader allocates only odd channels (starting with 1) and the Follower allocates only even channels (starting with 2). That is, the side producing the Offer first opens a subchannel and then puts the resulting ID into the Offer message. -There are two kinds of repies to an offer, either an Accept message or a Reject message. +There are two kinds of repies to an offer: either an Accept message or a Reject message. Reject messages look like this: ```python @@ -63,16 +77,16 @@ class OfferAccept: id: int # matching identifier for an existing offer from the other side ``` -When the offering side gets an `OfferReject` message, the subchannel is immediately closed. -The offering side may show the "reason" string to the user. -This offer ID should never be re-used during this session. +When the offering side gets an `OfferReject` message, the subchannel SHOULD be immediately closed. +The offering side MAY show the "reason" string to the user. +Any send Offer ID MUST NOT be re-used during this session. When the offering side gets an `OfferAccept` message it begins streaming the file over the already-opened subchannel. When completed, the subchannel is closed. That is, the offering side always initiates the open and close of the corresponding subchannel. -See examples down below. +See examples down below, after "Discussion". ## Discussion and Open Questions From efc0011e84f5aaa5dbd0f42a0e5f29d1dd2a443a Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 13 Jun 2022 13:50:51 -0600 Subject: [PATCH 03/49] overall selection --- file-transfer-v2.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/file-transfer-v2.md b/file-transfer-v2.md index afa41fa..9efd4ac 100644 --- a/file-transfer-v2.md +++ b/file-transfer-v2.md @@ -91,6 +91,13 @@ See examples down below, after "Discussion". ## Discussion and Open Questions +* overall selection of "classic" or "v2" file-transfer + +How will clients select between "classic" file-transfer support and Dilation-enabled v2 support? + +Perhaps it is sufficient to trigger that on whether Dilation negotiation worked or not: if we have a Dilation connection, then we do "v2 file-transfer" (and use sub-versions to select features there, see next open question). +If we do not, then "classic" / v1 file-transfer is used. + * (sub)versioning of this protocol It is likely useful to assign version numbers to either the protocol or to Offer messages. From 58e32e5a1f5c986d4b5bbf2d5a81878153fcf706 Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 13 Jun 2022 17:06:02 -0600 Subject: [PATCH 04/49] attempt at state diagram --- file-transfer-v2.dot | 70 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100644 file-transfer-v2.dot diff --git a/file-transfer-v2.dot b/file-transfer-v2.dot new file mode 100644 index 0000000..a506ac1 --- /dev/null +++ b/file-transfer-v2.dot @@ -0,0 +1,70 @@ +digraph { + label="File Transfer v2 state-machine"; + pack=true; + rankdir="TB"; + ranksep="1.0 equally"; + nodesep=1.5; +/*graph [nodesep=1.5];*/ +node[]; +edge [labelfloat=true,labelfontsize=16,]; + + start [style=bold,fontcolor=blue]; + + // the Dilation connection becomes available + start -> got_connect[arrowhead=none]; + got_connect -> await_version; + got_connect[shape=box,color=sienna,label="connected\lsend_version"] + + // user creates an Offer + start -> create_offer0[arrowhead=none]; + create_offer0 -> start; + create_offer0[shape=box,color=sienna,label="create:Offer\lqueue_offer()"]; + + // the other side sends us a Version message + await_version -> got_version[arrowhead=none]; + got_version -> connected; + got_version[shape=box,color=purple,label="got:Version\lsend_queued_offers()"]; + + // user creates an Offer + await_version -> create_offer1[arrowhead=none]; + create_offer1 -> await_version; + create_offer1[shape=box,color=sienna,label="create:Offer\lqueue_offer()"]; + + // other side sends offer, we notify our human + connected -> got_offer[arrowhead=none]; + got_offer -> connected; + got_offer[shape=box,color=purple,label="got:Offer\lnotify:offer_received"]; + + // other side rejects our offer, notify human + connected -> reject_offer[arrowhead=none]; + reject_offer -> connected; + reject_offer[shape=box,color=purple,label="got:Reject\lnotify:offer_rejected"]; + + // other side accepts our offer, send file + connected -> got_accept[arrowhead=none]; + got_accept -> connected; + got_accept[shape=box,color=purple,label="got:Accept\lsend_file()"]; + + // human tells us to stop, shut down + connected -> send_stop[arrowhead=none]; + send_stop -> closing; + send_stop[shape=box,color=sienna,label="stop\lclose_mailbox()"]; + + // user creates an Offer + connected -> create_offer2[arrowhead=none]; + create_offer2 -> connected; + create_offer2[shape=box,color=sienna,label="create:Offer\lsend_offer()"]; + + // our human accepts an offer, download the file + connected -> accept_offer[arrowhead=none]; + accept_offer -> connected; + accept_offer[shape=box,color=sienna,label="accept Offer\lreceive_file()"]; + + // mailbox confirms close + closing -> await_close[arrowhead=none]; + await_close -> done; + await_close[shape=box,color=purple,label="mailbox_closed\lnotify:finished"]; + + done [style=bold,fontcolor=blue]; + +} \ No newline at end of file From 8240b1a37ad3eb4e88ff7ffe36f66871e60809c9 Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 13 Jun 2022 22:13:26 -0600 Subject: [PATCH 05/49] tweak diagram --- file-transfer-v2.dot | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/file-transfer-v2.dot b/file-transfer-v2.dot index a506ac1..ff53384 100644 --- a/file-transfer-v2.dot +++ b/file-transfer-v2.dot @@ -1,5 +1,7 @@ digraph { label="File Transfer v2 state-machine"; + labelfontsize=40; + labelfontname="Source Code Pro"; pack=true; rankdir="TB"; ranksep="1.0 equally"; @@ -13,7 +15,7 @@ edge [labelfloat=true,labelfontsize=16,]; // the Dilation connection becomes available start -> got_connect[arrowhead=none]; got_connect -> await_version; - got_connect[shape=box,color=sienna,label="connected\lsend_version"] + got_connect[shape=box,color=sienna,label="connected\lsend_version()"] // user creates an Offer start -> create_offer0[arrowhead=none]; @@ -67,4 +69,34 @@ edge [labelfloat=true,labelfontsize=16,]; done [style=bold,fontcolor=blue]; + + // bunch of error cases .. technically there's probably a few + // more (like if we receive anything at all in "closing" or + // "closing_error") + // + // uncomment to .. clutter the diagram + + /* + + // get Offer before Version + await_version -> offer_err0[arrowhead=none,color=red]; + offer_err0 -> closing_error[color=red]; + offer_err0[shape=box,color=purple,label="got:Offer\lclose_mailbox()"] + + // get Accept before Version + await_version -> offer_ans0[arrowhead=none,color=red]; + offer_ans0 -> closing_error[color=red]; + offer_ans0[shape=box,color=purple,label="got:Accept\lclose_mailbox()"] + + // get Reject before Version + await_version -> offer_rej0[arrowhead=none,color=red]; + offer_rej0 -> closing_error[color=red]; + offer_rej0[shape=box,color=purple,label="got:Reject\lclose_mailbox()"] + + // closing_error worked + closing_error -> err_close[arrowhead=none,color=red]; + err_close -> done[color=red]; + err_close[shape=box,color=purple,label="mailbox_closed\lnotify:finished_error"]; + + */ } \ No newline at end of file From 8e8985df4ace78383bad055ee6f65ddd5ba6461c Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 13 Jun 2022 23:27:07 -0600 Subject: [PATCH 06/49] more tweaks --- file-transfer-v2.md | 61 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 58 insertions(+), 3 deletions(-) diff --git a/file-transfer-v2.md b/file-transfer-v2.md index 9efd4ac..2faea9f 100644 --- a/file-transfer-v2.md +++ b/file-transfer-v2.md @@ -6,16 +6,23 @@ Both sides must support and use Dilation (see `dilation-protocol.md`). Any all-caps words ("MAY", "MUST", etc) follow RFC2119 conventions. +NOTE: there are several open questions / discussion points, some with corresponding "XXX" comments inline. + ## Overview and Features -We describe a flexible, "session"-based approach to file transfer allowing either side to offer files to send (while the other side may accept or reject each offer). -Either side MAY terminate the transfer session. +We describe a flexible, "session"-based approach to file transfer allowing either side to offer files to send while the other side may accept or reject each offer. +Either side MAY terminate the transfer session (by closing the wormhole) -File are sent individually, with no dependency on zip or other archive formats. +File are offered and sent individually, with no dependency on zip or other archive formats. Metadata is included in the offers to allow the receiver to decide if they want that file before the transfer begins. +Filenames are relative paths. +When sending individual files, this will be simply the filename portion (with no leading paths). +For a series of files in a directory (i.e. if a directory was selected to send) paths will be relative to that directory (starting with the directory itself). +(XXX see "file naming" in discussion) + ## Protocol Details @@ -117,6 +124,54 @@ Preliminary conclusion: a simple Version message is sent first, with version=0. While `msgpack` is mentioned above, there are several other binary-supporting libraries worth considering. These are (in no particular order) at least: CBOR or flatbuffers or protocolbuffers or cap'n'proto +* file naming + +Sending a single file like `/home/meejah/Documents/Letter.docx` gets a filename like `Letter.docx` +Sending a whole directory like `/home/meejah/Documents/` would result in some number of offers like `Documents/Letter.docx` etc. + +The question is, does this make sense? +Should there (instead) be at least two kinds of offer: single files, or "collections"? +Maybe _all_ offers should be collections, and a "single file offer" is just the special case where the list of offers has a single entry. + + +## File Naming Example + +Given a hypothetical directory tree: + +* /home/ + * meejah/ + * grumpy-cat.jpeg + * homework-draft2-final.docx + * project/ + * local-network.dia + * traffic.pcap + * README + * src/ + * hello.py + +As spec'd above, if the human selects `/home/meejah/project/src/hello.py` then it should be sent as `hello.py`. +However if they select `/home/meejah/project/` then there should be 4 offers: project/local-network.dia`, `project/traffic.pcap`, `project/README`, `project/src/hello.py`. + +Another way to this could be to re-design Offer messages to look like this: + +```python +class Offer: + kind: int = 1 # "offer" + id: int # unique random identifier for this offer + path: str # utf8-encoded unicode relative base path of all files + files: list # contains FileOffer instances + +class FileOffer: + filename: str # utf8-encoded unicode relative pathname + timestamp: int # Unix timestamp (seconds since the epoch in GMT) + bytes: int # total number of bytes in the file + subchannel: int # the subchannel which the file will be sent on +``` + +This would keep collections of files together (e.g. a subdirectory). +For a single file, `path` would be `"."`. +Maybe: single `subchannel` for all files? (No need for framing / EOF; we have length) + ## Example 1 From 0e94de2fec47034366ea4f0fd64282efc29858b8 Mon Sep 17 00:00:00 2001 From: meejah Date: Wed, 15 Jun 2022 01:46:09 -0600 Subject: [PATCH 07/49] more version work --- file-transfer-v2.dot | 28 +++++-------- file-transfer-v2.md | 93 ++++++++++++++++++++++++++++++++++++++------ 2 files changed, 90 insertions(+), 31 deletions(-) diff --git a/file-transfer-v2.dot b/file-transfer-v2.dot index ff53384..61dc9c7 100644 --- a/file-transfer-v2.dot +++ b/file-transfer-v2.dot @@ -10,27 +10,17 @@ digraph { node[]; edge [labelfloat=true,labelfontsize=16,]; - start [style=bold,fontcolor=blue]; + await_dilation [style=bold,fontcolor=blue]; - // the Dilation connection becomes available - start -> got_connect[arrowhead=none]; - got_connect -> await_version; - got_connect[shape=box,color=sienna,label="connected\lsend_version()"] - - // user creates an Offer - start -> create_offer0[arrowhead=none]; - create_offer0 -> start; + // user creates an Offer (before connection has completed) + await_dilation -> create_offer0[arrowhead=none]; + create_offer0 -> await_dilation; create_offer0[shape=box,color=sienna,label="create:Offer\lqueue_offer()"]; - // the other side sends us a Version message - await_version -> got_version[arrowhead=none]; - got_version -> connected; - got_version[shape=box,color=purple,label="got:Version\lsend_queued_offers()"]; - - // user creates an Offer - await_version -> create_offer1[arrowhead=none]; - create_offer1 -> await_version; - create_offer1[shape=box,color=sienna,label="create:Offer\lqueue_offer()"]; + // the Dilation connection becomes available + await_dilation -> got_connect[arrowhead=none]; + got_connect -> connected; + got_connect[shape=box,color=sienna,label="connected\lsend_queued_offers()"] // other side sends offer, we notify our human connected -> got_offer[arrowhead=none]; @@ -52,7 +42,7 @@ edge [labelfloat=true,labelfontsize=16,]; send_stop -> closing; send_stop[shape=box,color=sienna,label="stop\lclose_mailbox()"]; - // user creates an Offer + // user creates an Offer (while connected) connected -> create_offer2[arrowhead=none]; create_offer2 -> connected; create_offer2[shape=box,color=sienna,label="create:Offer\lsend_offer()"]; diff --git a/file-transfer-v2.md b/file-transfer-v2.md index 2faea9f..a294f86 100644 --- a/file-transfer-v2.md +++ b/file-transfer-v2.md @@ -24,30 +24,60 @@ For a series of files in a directory (i.e. if a directory was selected to send) (XXX see "file naming" in discussion) +## Version Negotiation + +There is an existing file-transfer protocol which does not use Dilation. +Clients supporting newer versions of file-transfer (i.e. the one in this document) SHOULD offer backwards compatibility. + +In the mailbox protocol, applications can indicate version information. +The existing file-transfer protocol doesn't use this so the version information is empty (indicating "version 1"). +This protocol will include a dict like: + +```json +{ + "transfer-v2": { + "mode": "{send|receive|connect}", + "formats": [1] + } +} +``` + +The `mode` key indicates the desired mode of that peer. +It has one of three values: +* `"send"`: the peer will send a single file/text (similar to classic transfer protocol) +* `"receive"`: the peer will receive at most one file or text (the other side of the above) +* `"connect"`: the peer will send and receive zero or more files before closing the session + +Note that `send` and `receive` above will still use Dilation as all clients supporting this protocol must. +If a peer sends no version information at all, it will be using the classic protocol (and is thus using Transit and not Dilation for the peer-to-peer connection). + +The `formats` key is a list of message-formats understood by the peer. +This allows for existing messages like `Offer` to be extended, or for new message types to be added. +Peers MUST _accept_ messages for any formats they support. +Peers MUST only send messages for formats in the other side's list. +Only one format exists currently: `1`. +Future extensions to this protocol will document which format version any new attributes or messages belong to. + +See "Example of Protocol Expansion" below for discussion about adding new attributes. + + ## Protocol Details See the Dilation document for details on the Dilation setup procedure. Once a Dilation-supporting connection is open, we will have a "control" subchannel (subchannel #0). All offers MUST be sent over the control channel. -All answers (accept or reject) MUST also sent over the control channel. +All answers (accept or reject) MUST also be sent over the control channel. All control-channel messages MUST be encoded using `msgpack`. --> XXX: see "message encoding" discussion -Control-channel message formats are described using Python3 pseudo-code to illustrate the exact data types involved. +Control-channel message formats are described using Python pseudo-code to illustrate the exact data types involved. All control-channel messages contain an integer "kind" field describing the type of message. -Both sides MUST immediately send a Version message on the control channel (XXX see "(sub)versioning of this protocol" dicussion). -Version messages look like this: +XXX: Rejected idea: Version message, because we already do version negotiation via mailbox features. -```python -class Version: - kind: int = 0 # "version" - version: int = 1 # the only existing (sub)version of "file transfer v2" -``` - -Either side MAY send any number of Offer messages at any time. +Either side MAY send any number of Offer messages at any time after the connection is set up. They MUST first open a subchannel to receive the subchannel ID. Offer messages look like this: @@ -93,6 +123,15 @@ When completed, the subchannel is closed. That is, the offering side always initiates the open and close of the corresponding subchannel. +Each side may also send a free-form text message at any time. +These messages look like: + +```python +class Message: + message: str # unicode string + kind: int = 4 # "text message" +```` + See examples down below, after "Discussion". @@ -107,8 +146,10 @@ If we do not, then "classic" / v1 file-transfer is used. * (sub)versioning of this protocol -It is likely useful to assign version numbers to either the protocol or to Offer messages. +Might it be useful to assign version numbers to either the protocol (e.g. an initial Version message?) or to Offer messages. This would allow future extensions such as adding more metadata to the Offer (e.g. `thumbnail: bytes`). +Or, do we just use the existing versioning? +Or, do we just insist that clients ignore unknown keys in Offer/etc messages? (Allowing extensions, but if we want to _depend_ on those extras, it needs a whole new protocol version?) The simplest would be to version the entire protocol. If we versioned the Offer messages, it's not clear what should happen if an old client receives a too-new Offer. Presumably, reject it -- but it would be better to let the sender know that the client is old and to re-send an older style offer. @@ -173,6 +214,34 @@ For a single file, `path` would be `"."`. Maybe: single `subchannel` for all files? (No need for framing / EOF; we have length) +## Example of Protocol Expansion + +Let us suppose that at some point in the future we decide to add `thumbnail: bytes` to the `Offer` messages. +We assign this format `2`, so `Offers` become: + +```python +class Offer: + kind: int = 1 # "offer" + id: int + filename: str + timestamp: int + bytes: int + subchannel: int + thumbnail: bytes # introduced in format 2; PNG data +``` + +All existing implementations will have `"formats": [1]`. +A new thumbnail-supporting implementations will send `"formats": [1, 2]`. + +A new peer speaking to an old peer will never see `thumbnail` in the Offers, because the old peer sent `"formats": [1]` so the new peer knows not to inclue that attribute (and the old peer won't ever send it). + +Two new peers speaking will both send `"formats": [1, 2]` and so will both include (and know how to interpret) `"thumbnail"` attributes on `Offers`. + +Additinoally, a new peer that _doesn't want_ to see `"thumbnail"` data (e.g. it's a CLI client) can not include `2` in their `"formats"` list. + +XXX: perhaps the formats should be strings, like `["original", "thumbnail"]` for this example?? + + ## Example 1 Alice contacts Bian to transfer a single file. From c8aaa0d9c14d7696ea5b4cc4978ff4bf3c119001 Mon Sep 17 00:00:00 2001 From: meejah Date: Thu, 16 Jun 2022 18:19:45 -0600 Subject: [PATCH 08/49] random notes --- file-transfer-v2.md | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/file-transfer-v2.md b/file-transfer-v2.md index a294f86..89c7ec6 100644 --- a/file-transfer-v2.md +++ b/file-transfer-v2.md @@ -237,10 +237,38 @@ A new peer speaking to an old peer will never see `thumbnail` in the Offers, bec Two new peers speaking will both send `"formats": [1, 2]` and so will both include (and know how to interpret) `"thumbnail"` attributes on `Offers`. -Additinoally, a new peer that _doesn't want_ to see `"thumbnail"` data (e.g. it's a CLI client) can not include `2` in their `"formats"` list. +Additionally, a new peer that _doesn't want_ to see `"thumbnail"` data (e.g. it's a CLI client) can not include `2` in their `"formats"` list. XXX: perhaps the formats should be strings, like `["original", "thumbnail"]` for this example?? +XXX: warner: exercise, take some "possibly useful extensions" as an exercise (as above). (expensive to generate thumbnail, so worthless if you're going to ignore it). + +another exercise: a behavior, e.g the "permissions thing" (have to send back an answer, send-side has to wait) + +another exercise: compression (on the stream) ... maybe you don't speak zlib ... _could_ do it with a fallback style (try with compression, try without compression) + + +need exercise for "big change" which would need "transfer-v3" upgrade... + + + +"how fine-grained": zooko says you probably only want _one_ version identifier. + +"up to version 4" + +"between version 2 and 5" + +-> output is single scalar (e.g. "use version 4") + +-> leader/follower thing in Dilation (leader decides from the intersections) + +-> sending list/set of versions even if the output is a single value is because experimental versions can be introduced, but then invoked. + + . tls example: ... + + +warner likes this: could release "transfer-v3" for "big changes" and use the "formats" / feature-flags thing above for "tiny changes" + ## Example 1 From bb917fe6a5c1ddb68e17126b565f519a03a86793 Mon Sep 17 00:00:00 2001 From: meejah Date: Thu, 16 Jun 2022 18:20:20 -0600 Subject: [PATCH 09/49] new writeup, incorporating extensive discussion from brian warner --- dilated-file-transfer.md | 434 +++++++++++++++++++++++++++++++++++++++ file-transfer-v2.md | 379 ---------------------------------- 2 files changed, 434 insertions(+), 379 deletions(-) create mode 100644 dilated-file-transfer.md delete mode 100644 file-transfer-v2.md diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md new file mode 100644 index 0000000..7a2421c --- /dev/null +++ b/dilated-file-transfer.md @@ -0,0 +1,434 @@ +# Dilated File-Transfer Protocol + +This version of the file-transfer protocol is a complete replacement for the original (referred to as "classic") file-transfer protocol. + +Both sides must support and use Dilation (see `dilation-protocol.md`). + +Any all-caps words ("MAY", "MUST", etc) follow RFC2119 conventions. + +NOTE: there are several open questions / discussion points, some with corresponding "XXX" comments inline. + + +## Overview and Features + +We describe a flexible, "session"-based approach to file transfer allowing either side to offer files to send while the other side may accept or reject each offer. +Either side MAY terminate the transfer session (by closing the wormhole) +Either side MAY select a one-way version, similar to the classic protocol. + +File are offered and sent individually, with no dependency on zip or other archive formats. + +Metadata is included in the offers to allow the receiver to decide if they want that file before the transfer begins. + +"Offers" generally correspond to what a user might select; a single-file offer is possible but so is a directory. +In both cases, they are treated as "an offer" although a directory may consist of dozens or more individual files. + +Filenames are relative paths. +When sending individual files, this will be simply the filename portion (with no leading paths). +For a series of files in a directory (i.e. if a directory was selected to send) paths will be relative to that directory (starting with the directory itself). +(XXX see "file naming" in discussion) + + +## Version Negotiation + +There is an existing file-transfer protocol which does not use Dilation (called "classic" in this document). +Clients supporting newer versions of file-transfer (i.e. the one in this document) SHOULD offer backwards compatibility. + +In the mailbox protocol, applications can indicate version information. +The existing file-transfer protocol doesn't use this so the version information is empty (indicating "classic"). +This protocol will include a dict like: + +```json +{ + "transfer": { + "version": 1, + "mode": "{send|receive|connect}", + "features": ["basic"], + "permission": ["ask", "yes"] + } +} +``` + +The `version` key indicates the highest version of this Dilated File Transfer protocol that this peer understands. +There is currently only one version: `1`. + +The `mode` key indicates the desired mode of that peer. +It has one of three values: +* `"send"`: the peer will only send files (similar to classic transfer protocol) +* `"receive"`: the peer only receive files (the other side of the above) +* `"connect"`: the peer will send and receive zero or more files before closing the session + +Note that `send` and `receive` above will still use Dilation as all clients supporting this protocol must. +If a peer sends no version information at all, it will be using the classic protocol (and is thus using Transit and not Dilation for the peer-to-peer connection). + +The `"features"` key is a list of message-formats / features understood by the peer. +This allows for existing messages to be extended, or for new message types to be added. +Peers MUST _accept_ messages for any features they support. +Peers MUST only send messages for features in the other side's list. +Only one format exists currently: `"basic"`. + + XXX:: maybe just lean on "version" for now? + +See "Example of Protocol Expansion" below for discussion about adding new attributes (including when we might increment the `"version"` instead of adding a new `"feature"`). + +The `"permission"` key specifies how to proceed with offers. +Using mode `ask` tells the sender to pause after transmitting the first metadata message and await an answer from the peer before streaming data. +Using mode `yes` means the peer will accept all incoming transfers so the sender should not pause after the metadata (and instead immediately start sending data messages). +This cuts down latency for "one-way" transfers (see "Discussion") + + +## Protocol Details + +See the Dilation document for details on the Dilation setup procedure. +Once a Dilation-supporting connection is open, we will have a "control" subchannel (subchannel #0). +Either peer can also open additional subchannels. + +All control-channel messages are encoded using `msgpack`. + --> XXX: see "message encoding" discussion + +Control-channel message formats are described using Python pseudo-code to illustrate the exact data types involved. + +All control-channel messages contain an integer "kind" field describing the type of message. + +Rejected idea: Version message, because we already do version negotiation via mailbox features. + +Rejected idea: Offer/Answer messages via the control channel: we need to open a subchannel anyway; the subchannel-IDs are not intended to be part of the Dilation API. + + +### Control Channel Messages + +Each side MAY send a free-form text message at any time. +These messages look like: + +```python +class Message: + message: str # unicode string + kind: int = 1 # "text message" +``` + + +### Making an Offer + +Either side MAY send any number of Offer messages at any time after the connection is set up. +If the other peer specified `"mode": "send"` then this peer MUST NOT make any offers. + +To make an offer the peer opens a subchannel. +Recall from the Dilation specification that subchannels are _record_ pipes (not simple byte-streams). + +All records on the subchannel are bytes, where the first byte indicates the kind of message and the remaining bytes are kind-dependant. + +The following kinds of messages exist(indicated by the first byte): +* 1: msgpack-encoded `FileOffer` message +* 2: msgpack-encoded `DirectoryOffer` message +* 3: msgpack-encoded `OfferAccept` message +* 4: msgpack-encoded `OfferReejct` message +* 5: file data bytes + +All other byte values are reserved for future use and MUST NOT be used. +XXX: maybe spec [0, 128) as reserved, and [128, 255) for "experiments"? + +The first message sent on the new subchannel is one of two offer messages. + +To offer a single file (with message kind `1`): + +```python +class FileOffer: + filename: str # unicode relative pathname + timestamp: int # Unix timestamp (seconds since the epoch in GMT) + bytes: int # total number of bytes in the file +``` + +To offer a directory tree of many files (with message kind `2`): +```python +class DirectoryOffer: + base: str # unicode pathname of the root directory (i.e. what the user selected) + size: int # total number of bytes in _all_ files + files: list[str] # a list containing relative paths for each file +``` + +The filenames in the `"files"` list are unicode relative paths (relative to the `"base"` from the `DirectoryOffer` and NOT including that part. + +For example: + +```python +DirectoryOffer( + base="project", + size: 165, + files=["README", "src/hello.py"] + ] +) +``` + +This indicates an offer to send two files, one in `"project/README"` and the other in `"project/src/hello.py"`. + + +What happens next depends on the mode of the peer. + +If the peer has `"mode": "yes"` then this peer MUST immediately start sending content messages (see below). + +If the peer has `"mode": "ask"` then this peer MUST NOT send any more messages and instead await an incoming message. + +That incoming message MUST be one of two reply messages: `OfferAccept` or `OfferReject`. +These are indicated by the kind byte of that message being `3` or `4` (see list above). + +```python +class OfferReject: + reason: str # unicode string describing why the offer is rejected +``` + +Accept messages are blank (that is, they are a single byte: `4`). + +```python +class OfferAccept: + pass +``` + +When the offering side gets an `OfferReject` message, the subchannel SHOULD be immediately closed. +The offering side MAY show the "reason" string to the user. + +When the offering side gets an `OfferAccept` message it begins streaming the file over the already-opened subchannel. +When completed, the subchannel is closed. + +That is, the offering side always initiates the open and close of the corresponding subchannel. + +Messages of kind `5` ("file data bytes") consist solely of file data. +A single data message MUST NOT exceed 65536 (65KiB) inculding the single byte for "kind". +Applications are free to choose how to fragment the file data so long as no single message is bigger than 65536 bytes. +A good default to choose in 2022 is 16KiB (2^14 - 1 payload bytes) +XXX: what is a good default? Dilation doesn't give guidance either... + +When sending a `DirectoryOffer` each individual file is preceeded by a `FileOffer` message. However the rules about "maybe wait for reply" no longer exist; that is, all file data SHOULD be immediately sent. + +See examples down below, after "Discussion". + + +## Discussion and Open Questions + +* Overall versioning considerations + +Versioning is hard. + +The existing file-transfer protocol does not include versioning. +Luckily, the mailbox protocol _itself_ allows for "application versioning" messages; this gives us an "out" here, effectively letting us send a "T minus 1" message via the "application version" dict. + +We do not have another such escape hatch (i.e. "T minus 2"), so if we get it wrong (again) then we potentially have a painful upgrade path. + +Currently, a file-transfer peer that sends zero version data is assumed to be "classic". +A file-transfer peer supporting Dilation and this new protocol sends `"transfer": {...}` as per the "Version Negotiation" section above. +We include a `"version"` key in that dict so that this transfer protocol may be extended in the future (see "Protocol Expansion Exercises" below). + +Additionally, a `"features"` key is included in that information. +Although releated, this is somewhat orthogonal to "versions". +That is, a peer may _know how to parse_ some (newer) version of this protocol but may still wish to _not_ support (or use) a particular feature. + + +* message encoding + +While `msgpack` is mentioned above, there are several other binary-supporting libraries worth considering. +These are (in no particular order) at least: CBOR or flatbuffers or protocolbuffers or cap'n'proto + +We could also still decide to simply use JSON, as that is also unambiguous. +Although it enjoys widespread support, JSON suffers from a lack of 64-bit integers (because JavaScript only supports 2^53 integers) and doesn't support a byte-string type (forcing e.g. any binary data to be encded in base64 or similar). + +preliminary conclusion: msgpack. + + +* file naming + +Sending a single file like `/home/meejah/Documents/Letter.docx` gets a filename like `Letter.docx` +Sending a whole directory like `/home/meejah/Documents/` would result in a directory-offer with basedir `Documents` and some number of files (possibly with sub-paths). + +This does NOT offer a client the chance to select "this" and "that" from a Directory offer (however, see the "Protocol Expansion Execrises" section). + +Preliminary conclusion: centering around "the thing a human would select" (i.e. "a file" or "a directory") makes the most sense. + + +* streaming data + +There is no "finished" message. Maybe there should be? (e.g. the receiving side sends back a hash of the file to confirm it received it properly?) + +Does "re-using" the `FileOffer` as a kind of "header" when streaming `DirectoryOffer` contents make sense? +We need _something_ to indicate the next file etc + +Do the limits on message size make sense? Should "65KiB" be much smaller, potentially? +(Given that network conditions etc vary a lot, I think it makes sense for the _spec_ to be somewhat flexible here and "65k" doesn't seem very onerous for most modern devices / computers) + + +* compression + +It may make sense to do compression of files. +See "Protocol Expansion Exercises" for more discussion. + +Preliminary conclusion: no compression in this version. + + +## File Naming Example + +Given a hypothetical directory tree: + +* /home/ + * meejah/ + * grumpy-cat.jpeg + * homework-draft2-final.docx + * project/ + * local-network.dia + * traffic.pcap + * README + * src/ + * hello.py + +As spec'd above, if the human selects `/home/meejah/project/src/hello.py` then it should be sent as `hello.py`. +However if they select `/home/meejah/project/` then there should be a Directory Offer offers like: + +```python +DirectoryOffer( + base="project", + size=4444, + files=[ + "local-network.dia", + "traffic.pcap", + "README", + "src/hello.py" + ] +) +``` + + +## Protocol Expansion Exercises + +Here we present several scenarios for different kinds of protocol expansion. +The point here is to discuss the _kinds_ of expansions that might happen. +The examples here ARE NOT part of the spec and SHOULD NOT be implemented. + +That said, we believe they're realistic features that _could_ make sense in future protocol expansions. + + +### Thumbnails + +Let us suppose that at some point in the future we decide to add `thumbnail: bytes` to the `Offer` messages. +It is reasonable to imagine that some clients may not make use of this feature at all (e.g. CLI programs) and so work and bandwidth can be saved by not producing and sending them. + +This becomes a new `"feature"` in the protocol. +That is, the version information is upgraded to allow `"features": ["basic", "thumbnails"]`. + +Peers that do not understand (or do not _want_) thumbnails do not include that in their `"features"` list. +So, according to the protocol, these peers should never receive anything related to thumbnails. +Only if both peers include `"features": ["basic", "thumbnails"]` will they receive thumbnail-related information. + +The thumbnail feature itself could be implemented by expanding the `Offer` message: + +```python +class Offer: + filename: str + timestamp: int + bytes: int + thumbnail: bytes # introduced in "thumbnail" feature; PNG data +``` +A new peer speaking to an old peer will never see `thumbnail` in the Offers, because the old peer sent `"formats": ["basic"]` so the new peer knows not to inclue that attribute (and the old peer won't ever send it). + +Two new peers speaking will both send `"formats": ["basic", "thumbnails"]` and so will both include (and know how to interpret) `"thumbnail"` attributes on `Offers`. + +Additionally, a new peer that _doesn't want_ to see `"thumbnail"` data (e.g. it's a CLI client) can simply not include `"thumbnail"` in their `"formats"` list even if their protocol implementation knows about it. + + +### Finer Grained Permissions + +What if we decide we want to expand the "ask" behavior to sub-items in a Directory. + +As this affects the behavior of both the sender (who now has to wait more often) and the receiver (who now has to send a new message) this means an overall protocol version bump. + +(XXX _does_ it though? I think we could use `"formats"` here too...) + +So, this means that `"version": 2` would be the newest version. +Any peer that sends a version lower than this (i.e. existing `1`) will not send any fine-grained information (or "yes" messages). +Any peer who sees the other side at a version lower than `2` thus cannot use this behavior and has to prtend to be a version `1` peer. + +If both peers send `2` then they can both use the new behavior (still using the overall `"yes"` versus `"ask"` switch that exists now, probably). + + +### Compression + +Suppose that we decide to introduce compression. + +(XXX again, probably just "features"?) + + +### Big Change + +What is a sort of change that might actually _require_ us to use the `"version": 2` lever? + + +### How to Represent Overall Version + +It can be useful to send a "list of versions" you support even if the ultimate outcome of a "version negotiation" is a single scalar (of "maximum version"). + +Something to do with being able to release (and then revoke) particular (possibly "experimental") versions. + +There may be an example in the TLS history surrounding this. + +This means we might want `"version": [1, 2]` for example instead of `"version": 1` or `"version": 2` alone. + +XXX expand, find TLS example + + +## Example: one-way transfer + +Alice has two computers: `laptop` and `desktop`. +Alice wishes to transfer some number of files from `laptop` to `desktop`. + +Alice initiates a `wormhole receive --yes` on the `desktop` machine, indicating that it should receive multiple files and not ask permission (writing each one immediately). +This program prints a code and waits for protocol setup. + +Aliec uses a GUI program on `laptop` that allows drag-and-drop sending of files. +In this program, she enters the code that `desktop` printed out. + +Once the Dilated connection is establed, Alice can drop any number of files into the GUI application and they will be immediately written on the `desktop` without interaction. + +In this protocol, the `desktop` peer sends version information: + +```json +{ + "transfer": { + "version": 1, + "mode": "receive", + "features": ["basic"], + "permission": ["yes"] + } +} +``` + +The `laptop` peer then knows to not pause on its subchannels to await permission. +For each file that Alice drops, it: +* opens a subchannel +* sends the metadata (via a `FileOffer` / kind=`1` record) +* immediately starts sending data (via kind=`5` records) +* closes the subchannel + +On the `desktop` peer, it patiently waits for subchannels to open. +When a subchannel opens, it: +* it reads the first record and finds a `FileOffer`, opening a local file to stream to +* reads subsequent data records, writing them to the open file +* notices the subchannel close +* double-checks that the corrent number of payload bytes were received +* closes the file + +When the GUI application finished (e.g. Alice closes it) the mailbox is closed. +The `desktop` peer notices this and exits. + + +## Example: multi-directional transfer session + +Alice and Bob are on a video-call together. +They are collaborating and wish to share at least one file. + +Both open a GUI wormhole application. +Alice opens hers first, clicking "connect" to generate a code. +She tells Bob the code, and he enters it in his GUI. + +A Dilated channel is established, and both GUIs indicate they are ready to receive and/or send files. + +As Alice drops files into her GUI, Bob's side waits for confirmation. +Several files could be in this state. +Whenever Bob clicks "accept" on a file, his client answers with an `OfferAccept` message and Alice's client starts sending the file. +If Bob were to click "reject" then his client would answer with an `OfferReject` and Alice's client would close the subchannel. + +When they are done, the session is closed, the Dilation channel is stopped, the mailbox is closed and no more files may be transferred. diff --git a/file-transfer-v2.md b/file-transfer-v2.md deleted file mode 100644 index 89c7ec6..0000000 --- a/file-transfer-v2.md +++ /dev/null @@ -1,379 +0,0 @@ -# File-Transfer Protocol v2 - -This version 2 protocol is a complete replacement for the original (referred to as "v1") file-transfer protocol. - -Both sides must support and use Dilation (see `dilation-protocol.md`). - -Any all-caps words ("MAY", "MUST", etc) follow RFC2119 conventions. - -NOTE: there are several open questions / discussion points, some with corresponding "XXX" comments inline. - - -## Overview and Features - -We describe a flexible, "session"-based approach to file transfer allowing either side to offer files to send while the other side may accept or reject each offer. -Either side MAY terminate the transfer session (by closing the wormhole) - -File are offered and sent individually, with no dependency on zip or other archive formats. - -Metadata is included in the offers to allow the receiver to decide if they want that file before the transfer begins. - -Filenames are relative paths. -When sending individual files, this will be simply the filename portion (with no leading paths). -For a series of files in a directory (i.e. if a directory was selected to send) paths will be relative to that directory (starting with the directory itself). -(XXX see "file naming" in discussion) - - -## Version Negotiation - -There is an existing file-transfer protocol which does not use Dilation. -Clients supporting newer versions of file-transfer (i.e. the one in this document) SHOULD offer backwards compatibility. - -In the mailbox protocol, applications can indicate version information. -The existing file-transfer protocol doesn't use this so the version information is empty (indicating "version 1"). -This protocol will include a dict like: - -```json -{ - "transfer-v2": { - "mode": "{send|receive|connect}", - "formats": [1] - } -} -``` - -The `mode` key indicates the desired mode of that peer. -It has one of three values: -* `"send"`: the peer will send a single file/text (similar to classic transfer protocol) -* `"receive"`: the peer will receive at most one file or text (the other side of the above) -* `"connect"`: the peer will send and receive zero or more files before closing the session - -Note that `send` and `receive` above will still use Dilation as all clients supporting this protocol must. -If a peer sends no version information at all, it will be using the classic protocol (and is thus using Transit and not Dilation for the peer-to-peer connection). - -The `formats` key is a list of message-formats understood by the peer. -This allows for existing messages like `Offer` to be extended, or for new message types to be added. -Peers MUST _accept_ messages for any formats they support. -Peers MUST only send messages for formats in the other side's list. -Only one format exists currently: `1`. -Future extensions to this protocol will document which format version any new attributes or messages belong to. - -See "Example of Protocol Expansion" below for discussion about adding new attributes. - - -## Protocol Details - -See the Dilation document for details on the Dilation setup procedure. -Once a Dilation-supporting connection is open, we will have a "control" subchannel (subchannel #0). - -All offers MUST be sent over the control channel. -All answers (accept or reject) MUST also be sent over the control channel. -All control-channel messages MUST be encoded using `msgpack`. - --> XXX: see "message encoding" discussion - -Control-channel message formats are described using Python pseudo-code to illustrate the exact data types involved. - -All control-channel messages contain an integer "kind" field describing the type of message. - -XXX: Rejected idea: Version message, because we already do version negotiation via mailbox features. - -Either side MAY send any number of Offer messages at any time after the connection is set up. -They MUST first open a subchannel to receive the subchannel ID. -Offer messages look like this: - -```python -class Offer: - kind: int = 1 # "offer" - id: int # unique random identifier for this offer - filename: str # utf8-encoded unicode relative pathname - timestamp: int # Unix timestamp (seconds since the epoch in GMT) - bytes: int # total number of bytes in the file - subchannel: int # the subchannel which the file will be sent on -``` - -The `id` in an Offer MUST NOT match any other Offer from this side. -The subchannel in an Offer MUST NOT match any subchannel in any existing Offer from this side nor from the other side. -This latter constraint is enforced by the Dilation implementation: the Leader allocates only odd channels (starting with 1) and the Follower allocates only even channels (starting with 2). -That is, the side producing the Offer first opens a subchannel and then puts the resulting ID into the Offer message. - -There are two kinds of repies to an offer: either an Accept message or a Reject message. -Reject messages look like this: - -```python -class OfferReject: - kind: int = 2 # "offer reject" - id: int # matching identifier for an existing offer from the other side - reason: str # utf8-encoded unicode string describing why the offer is rejected -``` - -Accept messages look like this: - -```python -class OfferAccept: - kind: int = 3 # "offer accpet" - id: int # matching identifier for an existing offer from the other side -``` - -When the offering side gets an `OfferReject` message, the subchannel SHOULD be immediately closed. -The offering side MAY show the "reason" string to the user. -Any send Offer ID MUST NOT be re-used during this session. - -When the offering side gets an `OfferAccept` message it begins streaming the file over the already-opened subchannel. -When completed, the subchannel is closed. - -That is, the offering side always initiates the open and close of the corresponding subchannel. - -Each side may also send a free-form text message at any time. -These messages look like: - -```python -class Message: - message: str # unicode string - kind: int = 4 # "text message" -```` - -See examples down below, after "Discussion". - - -## Discussion and Open Questions - -* overall selection of "classic" or "v2" file-transfer - -How will clients select between "classic" file-transfer support and Dilation-enabled v2 support? - -Perhaps it is sufficient to trigger that on whether Dilation negotiation worked or not: if we have a Dilation connection, then we do "v2 file-transfer" (and use sub-versions to select features there, see next open question). -If we do not, then "classic" / v1 file-transfer is used. - -* (sub)versioning of this protocol - -Might it be useful to assign version numbers to either the protocol (e.g. an initial Version message?) or to Offer messages. -This would allow future extensions such as adding more metadata to the Offer (e.g. `thumbnail: bytes`). -Or, do we just use the existing versioning? -Or, do we just insist that clients ignore unknown keys in Offer/etc messages? (Allowing extensions, but if we want to _depend_ on those extras, it needs a whole new protocol version?) - -The simplest would be to version the entire protocol. -If we versioned the Offer messages, it's not clear what should happen if an old client receives a too-new Offer. Presumably, reject it -- but it would be better to let the sender know that the client is old and to re-send an older style offer. - -To version the entire protocol we'd have to have each side send an initial control-channel message indicating what version they support. -Perhaps this could be stated as "features" instead (e.g. "thumbnail" feature if we added that, or "extended metadata" if we add more metadata, etc). -Using "features" would have the added semantic benefit that two up-to-date clients may still not want (or be able to use) particular features (e.g. a CLI client might not display thumbnails) but is more complex for both sides. - -Preliminary conclusion: a simple Version message is sent first, with version=0. - -* message encoding - -While `msgpack` is mentioned above, there are several other binary-supporting libraries worth considering. -These are (in no particular order) at least: CBOR or flatbuffers or protocolbuffers or cap'n'proto - -* file naming - -Sending a single file like `/home/meejah/Documents/Letter.docx` gets a filename like `Letter.docx` -Sending a whole directory like `/home/meejah/Documents/` would result in some number of offers like `Documents/Letter.docx` etc. - -The question is, does this make sense? -Should there (instead) be at least two kinds of offer: single files, or "collections"? -Maybe _all_ offers should be collections, and a "single file offer" is just the special case where the list of offers has a single entry. - - -## File Naming Example - -Given a hypothetical directory tree: - -* /home/ - * meejah/ - * grumpy-cat.jpeg - * homework-draft2-final.docx - * project/ - * local-network.dia - * traffic.pcap - * README - * src/ - * hello.py - -As spec'd above, if the human selects `/home/meejah/project/src/hello.py` then it should be sent as `hello.py`. -However if they select `/home/meejah/project/` then there should be 4 offers: project/local-network.dia`, `project/traffic.pcap`, `project/README`, `project/src/hello.py`. - -Another way to this could be to re-design Offer messages to look like this: - -```python -class Offer: - kind: int = 1 # "offer" - id: int # unique random identifier for this offer - path: str # utf8-encoded unicode relative base path of all files - files: list # contains FileOffer instances - -class FileOffer: - filename: str # utf8-encoded unicode relative pathname - timestamp: int # Unix timestamp (seconds since the epoch in GMT) - bytes: int # total number of bytes in the file - subchannel: int # the subchannel which the file will be sent on -``` - -This would keep collections of files together (e.g. a subdirectory). -For a single file, `path` would be `"."`. -Maybe: single `subchannel` for all files? (No need for framing / EOF; we have length) - - -## Example of Protocol Expansion - -Let us suppose that at some point in the future we decide to add `thumbnail: bytes` to the `Offer` messages. -We assign this format `2`, so `Offers` become: - -```python -class Offer: - kind: int = 1 # "offer" - id: int - filename: str - timestamp: int - bytes: int - subchannel: int - thumbnail: bytes # introduced in format 2; PNG data -``` - -All existing implementations will have `"formats": [1]`. -A new thumbnail-supporting implementations will send `"formats": [1, 2]`. - -A new peer speaking to an old peer will never see `thumbnail` in the Offers, because the old peer sent `"formats": [1]` so the new peer knows not to inclue that attribute (and the old peer won't ever send it). - -Two new peers speaking will both send `"formats": [1, 2]` and so will both include (and know how to interpret) `"thumbnail"` attributes on `Offers`. - -Additionally, a new peer that _doesn't want_ to see `"thumbnail"` data (e.g. it's a CLI client) can not include `2` in their `"formats"` list. - -XXX: perhaps the formats should be strings, like `["original", "thumbnail"]` for this example?? - -XXX: warner: exercise, take some "possibly useful extensions" as an exercise (as above). (expensive to generate thumbnail, so worthless if you're going to ignore it). - -another exercise: a behavior, e.g the "permissions thing" (have to send back an answer, send-side has to wait) - -another exercise: compression (on the stream) ... maybe you don't speak zlib ... _could_ do it with a fallback style (try with compression, try without compression) - - -need exercise for "big change" which would need "transfer-v3" upgrade... - - - -"how fine-grained": zooko says you probably only want _one_ version identifier. - -"up to version 4" - -"between version 2 and 5" - --> output is single scalar (e.g. "use version 4") - --> leader/follower thing in Dilation (leader decides from the intersections) - --> sending list/set of versions even if the output is a single value is because experimental versions can be introduced, but then invoked. - - . tls example: ... - - -warner likes this: could release "transfer-v3" for "big changes" and use the "formats" / feature-flags thing above for "tiny changes" - - -## Example 1 - -Alice contacts Bian to transfer a single file. - -* The software on Alice's computer begins a Dilation-enabled session, producing a secret code. -* Alice sends this code to Bian -* Software on Bian's computer uses the code to complete the Dilation-enabled session. - -At this point, Alice and Bian are connected (possibly directly, possibly via the relay). -Alice is the "Leader" in the Dilation protocol. -From this point on, the "file transfer v2" protocol is spoken (see below for `seqdiag` markup to render a diagarm). - -* Alice opens a new subchannel (id=1, because she's the Leader) -* Alice sends an Offer to Bian on the control channel -* Bian accepts the Offer -* Alice sends all data on subchannel 1 -* Alice closes subchannel 1 -* Bian closes the mailbox -* Alice also closes the mailbox (which is now de-allocated on the server) - -Here is a sequence diagram of the above. - -```seqdiag -seqdiag { - Alice -> Bian [label="OPEN(subchannel=1)"] - Alice -> Bian [label="control \n Offer[filename=foo, subchannel=1, id=42]"] - - Alice <- Bian [label="control \n Accept[id=42]"] - - Alice -> Bian [label="subchannel 1 \n DATA"] - Alice -> Bian [label="subchannel 1 \n DATA"] - Alice -> Bian [label="subchannel 1 \n DATA"] - - Alice -> Bian [label="CLOSE(subchannel=1)"] - - Alice -> Bian [label="close mailbox"] -} -``` - -## Example 2 - -Alice contacts Bian to start a file-transfer session, sending 2 files and receiving 1. - -The software is started on Alice's computer which initiates a (Dilation-enabled) connection. -Alice communcates the secret code to Bian. -On Bian's computer, the (Dilation-enabled) software completes the connection. - -So, we have a Dilation-enabled connection between Alice and Bian's computers. - -```seqdiag -seqdiag { - Alice -> Bian [label="OPEN(subchannel=1)"] - Alice -> Bian [label="control \n Offer[filename=foo, subchannel=1, id=42]"] - Alice -> Bian [label="OPEN(subchannel=3)"] - Alice -> Bian [label="control \n Offer[filename=bar, subchannel=3, id=89]"] - - Alice <- Bian [label="control \n Accept[id=42]"] - Alice -> Bian [label="subchannel 1 \n DATA"] - - Alice <- Bian [label="OPEN(subchannel=2)"] - Alice <- Bian [label="control \n Offer[filename=quux, subchannel=2, id=65]"] - - Alice <- Bian [label="control \n Accept[id=89]"] - Alice -> Bian [label="subchannel 3 \n DATA"] - Alice -> Bian [label="subchannel 1 \n DATA"] - Alice -> Bian [label="subchannel 3 \n DATA"] - Alice -> Bian [label="subchannel 1 \n DATA"] - - Alice -> Bian [label="control \n Accept[id=65]"] - Alice <- Bian [label="subchannel 2 \n DATA"] - - Alice -> Bian [label="CLOSE(subchannel=1)"] - - Alice <- Bian [label="subchannel 2 \n DATA"] - Alice -> Bian [label="subchannel 3 \n DATA"] - Alice -> Bian [label="subchannel 3 \n DATA"] - - Alice <- Bian [label="CLOSE(subchannel=2)"] - Alice -> Bian [label="CLOSE(subchannel=3)"] - - Alice -> Bian [label="close mailbox"] - Alice <- Bian [label="close mailbox"] -} -``` - -Breaking down all those messages at a high level this is what's happening: - -* Alice is the "Leader" in the Dilation session (this affects the subchannel IDs). -* Alice opens subchannel 1 to allocate an id to send in the first Offer -* Alice sends the first Offer, of file `foo`. -* Alice opens subchannel 3 for the second Offer -* Alice sends the second Offer, of file `bar` -* Bian accepts the Offer for `foo`. -* Alice starts sending data for `foo` on subchannel 1. -* Bian opens subchannel 2 -* Bian sends Offer of file `quux` -* Bian accepts Offer 89, for file `bar` -* Alice sends more data (2 packets each of `foo` and `bar` files) -* Alice accepts Offer 65, for file `quux` -* Bian sends first data of file `quux` -* Alice is done sending file `foo` and closes the subchannel -* Bian sends some data for `quux` -* Alice sends 2 more chunks of data for `bar` -* Bian is done sending `quux` and closes the subchannel -* Alice is done sending `bar` and closes the subchannel -* Alice indicates she is done with the session and closes the mailbox -* Bian acknowledges and also closes the mailbox From 75d1c05240efd53db62ebeb8008b02919e475367 Mon Sep 17 00:00:00 2001 From: meejah Date: Fri, 17 Jun 2022 00:47:13 -0600 Subject: [PATCH 10/49] clarify --- dilated-file-transfer.md | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 7a2421c..b5c0791 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -43,7 +43,7 @@ This protocol will include a dict like: "version": 1, "mode": "{send|receive|connect}", "features": ["basic"], - "permission": ["ask", "yes"] + "permission": "{ask|yes}" } } ``` @@ -378,12 +378,12 @@ Alice wishes to transfer some number of files from `laptop` to `desktop`. Alice initiates a `wormhole receive --yes` on the `desktop` machine, indicating that it should receive multiple files and not ask permission (writing each one immediately). This program prints a code and waits for protocol setup. -Aliec uses a GUI program on `laptop` that allows drag-and-drop sending of files. +Alice uses a GUI program on `laptop` that allows drag-and-drop sending of files. In this program, she enters the code that `desktop` printed out. Once the Dilated connection is establed, Alice can drop any number of files into the GUI application and they will be immediately written on the `desktop` without interaction. -In this protocol, the `desktop` peer sends version information: +Speaking this protocol, the `desktop` (receive-only CLI) peer sends version information: ```json { @@ -391,27 +391,28 @@ In this protocol, the `desktop` peer sends version information: "version": 1, "mode": "receive", "features": ["basic"], - "permission": ["yes"] + "permission": "yes" } } ``` -The `laptop` peer then knows to not pause on its subchannels to await permission. -For each file that Alice drops, it: +The `laptop` peer then knows to not pause on its subchannels to await permission (`"permission": "yes"`). +For each file that Alice drops, the `laptop` peer: * opens a subchannel -* sends the metadata (via a `FileOffer` / kind=`1` record) +* sends a `FileOffer` / kind=`1` record * immediately starts sending data (via kind=`5` records) -* closes the subchannel +* closes the subchannel (when all data is sent) -On the `desktop` peer, it patiently waits for subchannels to open. +On the `desktop` peer, the program patiently waits for subchannels to open. When a subchannel opens, it: -* it reads the first record and finds a `FileOffer`, opening a local file to stream to +* reads the first record and finds a `FileOffer` and opens a local file for writing * reads subsequent data records, writing them to the open file * notices the subchannel close * double-checks that the corrent number of payload bytes were received + * XXX: return a checksum ack? (this might be more in line with Waterken principals so the sending side knows to delete state relating to this file ... but arguably with Dilation it "knows" that file made it?) * closes the file -When the GUI application finished (e.g. Alice closes it) the mailbox is closed. +When the GUI application finishes (e.g. Alice closes it) the mailbox is closed. The `desktop` peer notices this and exits. @@ -428,7 +429,11 @@ A Dilated channel is established, and both GUIs indicate they are ready to recei As Alice drops files into her GUI, Bob's side waits for confirmation. Several files could be in this state. -Whenever Bob clicks "accept" on a file, his client answers with an `OfferAccept` message and Alice's client starts sending the file. +Whenever Bob clicks "accept" on a file, his client answers with an `OfferAccept` message and Alice's client starts sending data records (the content of the file). If Bob were to click "reject" then his client would answer with an `OfferReject` and Alice's client would close the subchannel. +XXX what if Bob is bored and clicks "cancel" on a file? -When they are done, the session is closed, the Dilation channel is stopped, the mailbox is closed and no more files may be transferred. +Alice and Bob may exchange severl files at different times in either direction. +As they wrap up the call, Bob close his GUI client which closes the mailbox (and Dilated connection). +Alice's client sees the mailbox close. +Alice's GUI tells her that Bob is done and finishes the session; she can no longer drop or add files. From 7caca5b17a20f4f7cc66b7f13c141dc3207fbc4b Mon Sep 17 00:00:00 2001 From: meejah Date: Fri, 17 Jun 2022 00:52:21 -0600 Subject: [PATCH 11/49] words --- dilated-file-transfer.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index b5c0791..efbd985 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -427,13 +427,13 @@ She tells Bob the code, and he enters it in his GUI. A Dilated channel is established, and both GUIs indicate they are ready to receive and/or send files. -As Alice drops files into her GUI, Bob's side waits for confirmation. +As Alice drops files into her GUI, Bob's side waits for confirmation from Bob. Several files could be in this state. Whenever Bob clicks "accept" on a file, his client answers with an `OfferAccept` message and Alice's client starts sending data records (the content of the file). -If Bob were to click "reject" then his client would answer with an `OfferReject` and Alice's client would close the subchannel. +Whenever Bob clicks "reject", his client answesr with an `OfferReject` and Alice's client closes the subchannel. XXX what if Bob is bored and clicks "cancel" on a file? -Alice and Bob may exchange severl files at different times in either direction. -As they wrap up the call, Bob close his GUI client which closes the mailbox (and Dilated connection). +Alice and Bob may exchange several files at different times, with either Alice or Bob being the sender. +As they wrap up the call, Bob closes his GUI client which closes the mailbox (and Dilated connection). Alice's client sees the mailbox close. Alice's GUI tells her that Bob is done and finishes the session; she can no longer drop or add files. From 545a0506c495f55bda0695d484315cbfdb71b511 Mon Sep 17 00:00:00 2001 From: meejah Date: Tue, 21 Jun 2022 14:10:48 -0600 Subject: [PATCH 12/49] basic -> core --- dilated-file-transfer.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index efbd985..b029f64 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -42,7 +42,7 @@ This protocol will include a dict like: "transfer": { "version": 1, "mode": "{send|receive|connect}", - "features": ["basic"], + "features": ["core"], "permission": "{ask|yes}" } } @@ -64,7 +64,7 @@ The `"features"` key is a list of message-formats / features understood by the p This allows for existing messages to be extended, or for new message types to be added. Peers MUST _accept_ messages for any features they support. Peers MUST only send messages for features in the other side's list. -Only one format exists currently: `"basic"`. +Only one format exists currently: `"core"`. XXX:: maybe just lean on "version" for now? @@ -390,7 +390,7 @@ Speaking this protocol, the `desktop` (receive-only CLI) peer sends version info "transfer": { "version": 1, "mode": "receive", - "features": ["basic"], + "features": ["core"], "permission": "yes" } } From efdf7eede17376c2ffaf68e9298333a03ebd2472 Mon Sep 17 00:00:00 2001 From: meejah Date: Tue, 21 Jun 2022 14:11:08 -0600 Subject: [PATCH 13/49] paths are lists; cleanups/edits --- dilated-file-transfer.md | 106 ++++++++++++++++++++++----------------- 1 file changed, 59 insertions(+), 47 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index b029f64..859558e 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -51,10 +51,10 @@ This protocol will include a dict like: The `version` key indicates the highest version of this Dilated File Transfer protocol that this peer understands. There is currently only one version: `1`. -The `mode` key indicates the desired mode of that peer. +The `mode` key indicates the desired mode of the peer. It has one of three values: * `"send"`: the peer will only send files (similar to classic transfer protocol) -* `"receive"`: the peer only receive files (the other side of the above) +* `"receive"`: the peer only receive files (the flip side of the above) * `"connect"`: the peer will send and receive zero or more files before closing the session Note that `send` and `receive` above will still use Dilation as all clients supporting this protocol must. @@ -66,13 +66,13 @@ Peers MUST _accept_ messages for any features they support. Peers MUST only send messages for features in the other side's list. Only one format exists currently: `"core"`. - XXX:: maybe just lean on "version" for now? + XXX:: maybe just lean on "version" for now? e.g. version `2` could introduce "features" See "Example of Protocol Expansion" below for discussion about adding new attributes (including when we might increment the `"version"` instead of adding a new `"feature"`). The `"permission"` key specifies how to proceed with offers. -Using mode `ask` tells the sender to pause after transmitting the first metadata message and await an answer from the peer before streaming data. -Using mode `yes` means the peer will accept all incoming transfers so the sender should not pause after the metadata (and instead immediately start sending data messages). +Using mode `"ask"` tells the sender to pause after transmitting the first metadata message and await an answer from the peer before streaming further data. +Using mode `"yes"` means the peer will accept all incoming transfers so the sender should not pause after the metadata (and instead immediately start sending data messages). This cuts down latency for "one-way" transfers (see "Discussion") @@ -85,9 +85,9 @@ Either peer can also open additional subchannels. All control-channel messages are encoded using `msgpack`. --> XXX: see "message encoding" discussion -Control-channel message formats are described using Python pseudo-code to illustrate the exact data types involved. +Control-channel message formats are described using Python pseudo-code to illustrate the data types involved. -All control-channel messages contain an integer "kind" field describing the type of message. +All control-channel messages contain an integer "kind" field describing the sort of message it is. Rejected idea: Version message, because we already do version negotiation via mailbox features. @@ -108,44 +108,51 @@ class Message: ### Making an Offer -Either side MAY send any number of Offer messages at any time after the connection is set up. -If the other peer specified `"mode": "send"` then this peer MUST NOT make any offers. +Either side MAY propose any number of Offers at any time after the connection is set up. +If the other peer specified `"mode": "send"` then this peer MUST NOT make any Offers. -To make an offer the peer opens a subchannel. +To make an Offer the peer opens a subchannel. Recall from the Dilation specification that subchannels are _record_ pipes (not simple byte-streams). -All records on the subchannel are bytes, where the first byte indicates the kind of message and the remaining bytes are kind-dependant. +All records on the subchannel are bytes, where the first byte indicates the kind of message and the remaining bytes are a kind-dependant payload. -The following kinds of messages exist(indicated by the first byte): +The following kinds of messages exist (as indicated by the first byte): * 1: msgpack-encoded `FileOffer` message * 2: msgpack-encoded `DirectoryOffer` message * 3: msgpack-encoded `OfferAccept` message -* 4: msgpack-encoded `OfferReejct` message +* 4: msgpack-encoded `OfferReject` message * 5: file data bytes All other byte values are reserved for future use and MUST NOT be used. -XXX: maybe spec [0, 128) as reserved, and [128, 255) for "experiments"? -The first message sent on the new subchannel is one of two offer messages. + XXX: maybe spec [0, 128) as reserved, and [128, 255) for "experiments"? + +The first message sent on the new subchannel is one of two sorts of offer messages. To offer a single file (with message kind `1`): ```python class FileOffer: - filename: str # unicode relative pathname - timestamp: int # Unix timestamp (seconds since the epoch in GMT) - bytes: int # total number of bytes in the file + filename: str # filename (no path segments) + timestamp: int # Unix timestamp (seconds since the epoch in GMT) + bytes: int # total number of bytes in the file ``` To offer a directory tree of many files (with message kind `2`): + ```python class DirectoryOffer: - base: str # unicode pathname of the root directory (i.e. what the user selected) - size: int # total number of bytes in _all_ files - files: list[str] # a list containing relative paths for each file + base: str # unicode pathname of the root directory (i.e. what the user selected) + size: int # total number of bytes in _all_ files + files: list[list[str]] # a list containing relative paths for each file + # each relative path is a sequence of unicode strings (relative to "base") ``` -The filenames in the `"files"` list are unicode relative paths (relative to the `"base"` from the `DirectoryOffer` and NOT including that part. +The filenames in the `"files"` list are sequences of unicode path-names and are relative to the `"base"` from the `DirectoryOffer` (but NOT including that part). +Note that a `FileOffer` message also preceeds each file in the Directory when the data is streamed. +The files MUST be streamed in the same order they appear in the `files` list. +The last segment of each entry in the filename list MUST match the `.filename` of the `FileOffer` message. + For example: @@ -153,13 +160,14 @@ For example: DirectoryOffer( base="project", size: 165, - files=["README", "src/hello.py"] + files=[ + ["README"], + ["src", "hello.py"], ] ) ``` -This indicates an offer to send two files, one in `"project/README"` and the other in `"project/src/hello.py"`. - +This indicates an offer to send a directory consisting of two files: one in `"project/README"` and the other in `"project/src/hello.py"`. What happens next depends on the mode of the peer. @@ -175,7 +183,7 @@ class OfferReject: reason: str # unicode string describing why the offer is rejected ``` -Accept messages are blank (that is, they are a single byte: `4`). +Accept messages are blank (that is, they have no payload, just the `kind` byte of `3`). ```python class OfferAccept: @@ -194,7 +202,8 @@ Messages of kind `5` ("file data bytes") consist solely of file data. A single data message MUST NOT exceed 65536 (65KiB) inculding the single byte for "kind". Applications are free to choose how to fragment the file data so long as no single message is bigger than 65536 bytes. A good default to choose in 2022 is 16KiB (2^14 - 1 payload bytes) -XXX: what is a good default? Dilation doesn't give guidance either... + + XXX: what is a good default? Dilation doesn't give guidance either... When sending a `DirectoryOffer` each individual file is preceeded by a `FileOffer` message. However the rules about "maybe wait for reply" no longer exist; that is, all file data SHOULD be immediately sent. @@ -284,10 +293,10 @@ DirectoryOffer( base="project", size=4444, files=[ - "local-network.dia", - "traffic.pcap", - "README", - "src/hello.py" + ["local-network.dia"], + ["traffic.pcap"], + ["README"], + ["src", "hello.py"], ] ) ``` @@ -304,43 +313,43 @@ That said, we believe they're realistic features that _could_ make sense in futu ### Thumbnails -Let us suppose that at some point in the future we decide to add `thumbnail: bytes` to the `Offer` messages. +Let us suppose we decide to add `thumbnail: bytes` to the `Offer` messages. It is reasonable to imagine that some clients may not make use of this feature at all (e.g. CLI programs) and so work and bandwidth can be saved by not producing and sending them. This becomes a new `"feature"` in the protocol. -That is, the version information is upgraded to allow `"features": ["basic", "thumbnails"]`. +That is, the version information is upgraded to allow `"features": ["core", "thumbnails"]`. Peers that do not understand (or do not _want_) thumbnails do not include that in their `"features"` list. So, according to the protocol, these peers should never receive anything related to thumbnails. -Only if both peers include `"features": ["basic", "thumbnails"]` will they receive thumbnail-related information. +Only if both peers include `"features": ["core", "thumbnails"]` will they receive thumbnail-related information. The thumbnail feature itself could be implemented by expanding the `Offer` message: ```python -class Offer: +class FileOffer: filename: str timestamp: int bytes: int thumbnail: bytes # introduced in "thumbnail" feature; PNG data ``` -A new peer speaking to an old peer will never see `thumbnail` in the Offers, because the old peer sent `"formats": ["basic"]` so the new peer knows not to inclue that attribute (and the old peer won't ever send it). +A new peer speaking to an old peer will never see `thumbnail` in the Offers, because the old peer sent `"formats": ["core"]` so the new peer knows not to inclue that attribute (and the old peer won't ever send it). -Two new peers speaking will both send `"formats": ["basic", "thumbnails"]` and so will both include (and know how to interpret) `"thumbnail"` attributes on `Offers`. +Two new peers speaking will both send `"formats": ["core", "thumbnails"]` and so will both include (and know how to interpret) `"thumbnail"` attributes on `Offers`. Additionally, a new peer that _doesn't want_ to see `"thumbnail"` data (e.g. it's a CLI client) can simply not include `"thumbnail"` in their `"formats"` list even if their protocol implementation knows about it. ### Finer Grained Permissions -What if we decide we want to expand the "ask" behavior to sub-items in a Directory. +What if we decide we want to expand the "ask" behavior to sub-items in a DirectoryOffer. As this affects the behavior of both the sender (who now has to wait more often) and the receiver (who now has to send a new message) this means an overall protocol version bump. -(XXX _does_ it though? I think we could use `"formats"` here too...) + XXX: _does_ it though? I think we could use `"formats"` here too... So, this means that `"version": 2` would be the newest version. -Any peer that sends a version lower than this (i.e. existing `1`) will not send any fine-grained information (or "yes" messages). -Any peer who sees the other side at a version lower than `2` thus cannot use this behavior and has to prtend to be a version `1` peer. +Any peer that sends a version lower than this (i.e. the existing `1`) will not send any fine-grained information (or "yes" messages). +Any peer who sees the other side at a version lower than `2` thus cannot use this behavior and has to pretend to be a version `1` peer. If both peers send `2` then they can both use the new behavior (still using the overall `"yes"` versus `"ask"` switch that exists now, probably). @@ -349,7 +358,7 @@ If both peers send `2` then they can both use the new behavior (still using the Suppose that we decide to introduce compression. -(XXX again, probably just "features"?) +(XXX again, probably just leverage "features"?) ### Big Change @@ -367,7 +376,7 @@ There may be an example in the TLS history surrounding this. This means we might want `"version": [1, 2]` for example instead of `"version": 1` or `"version": 2` alone. -XXX expand, find TLS example + XXX expand, find TLS example ## Example: one-way transfer @@ -397,15 +406,17 @@ Speaking this protocol, the `desktop` (receive-only CLI) peer sends version info ``` The `laptop` peer then knows to not pause on its subchannels to await permission (`"permission": "yes"`). +This saves one round-trip per file. For each file that Alice drops, the `laptop` peer: * opens a subchannel * sends a `FileOffer` / kind=`1` record * immediately starts sending data (via kind=`5` records) * closes the subchannel (when all data is sent) -On the `desktop` peer, the program patiently waits for subchannels to open. +On the `desktop` peer, the program waits for subchannels to open. When a subchannel opens, it: -* reads the first record and finds a `FileOffer` and opens a local file for writing +* reads the first record +* finds a `FileOffer` and opens a local file for writing * reads subsequent data records, writing them to the open file * notices the subchannel close * double-checks that the corrent number of payload bytes were received @@ -430,8 +441,9 @@ A Dilated channel is established, and both GUIs indicate they are ready to recei As Alice drops files into her GUI, Bob's side waits for confirmation from Bob. Several files could be in this state. Whenever Bob clicks "accept" on a file, his client answers with an `OfferAccept` message and Alice's client starts sending data records (the content of the file). -Whenever Bob clicks "reject", his client answesr with an `OfferReject` and Alice's client closes the subchannel. -XXX what if Bob is bored and clicks "cancel" on a file? +Whenever Bob clicks "reject", his client answers with an `OfferReject` and Alice's client closes the subchannel. + +XXX what if Bob gets bored and clicks "cancel" on a file? Alice and Bob may exchange several files at different times, with either Alice or Bob being the sender. As they wrap up the call, Bob closes his GUI client which closes the mailbox (and Dilated connection). From 12eacc9be2b92101ae347f309fd733f0c14a5359 Mon Sep 17 00:00:00 2001 From: meejah Date: Tue, 21 Jun 2022 14:11:45 -0600 Subject: [PATCH 14/49] clean --- dilated-file-transfer.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 859558e..a4682b8 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -6,7 +6,7 @@ Both sides must support and use Dilation (see `dilation-protocol.md`). Any all-caps words ("MAY", "MUST", etc) follow RFC2119 conventions. -NOTE: there are several open questions / discussion points, some with corresponding "XXX" comments inline. + NOTE: there are several open questions / discussion points, some with corresponding "XXX" comments inline. ## Overview and Features @@ -15,7 +15,7 @@ We describe a flexible, "session"-based approach to file transfer allowing eithe Either side MAY terminate the transfer session (by closing the wormhole) Either side MAY select a one-way version, similar to the classic protocol. -File are offered and sent individually, with no dependency on zip or other archive formats. +Files are offered and sent individually, with no dependency on zip or other archive formats. Metadata is included in the offers to allow the receiver to decide if they want that file before the transfer begins. From 23e1e8abb285a09131154cb45a3814d8782969b8 Mon Sep 17 00:00:00 2001 From: meejah Date: Tue, 21 Jun 2022 14:23:54 -0600 Subject: [PATCH 15/49] clean --- dilated-file-transfer.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index a4682b8..72860be 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -13,7 +13,7 @@ Any all-caps words ("MAY", "MUST", etc) follow RFC2119 conventions. We describe a flexible, "session"-based approach to file transfer allowing either side to offer files to send while the other side may accept or reject each offer. Either side MAY terminate the transfer session (by closing the wormhole) -Either side MAY select a one-way version, similar to the classic protocol. +Either side MAY select a one-way mode, similar to the classic protocol. Files are offered and sent individually, with no dependency on zip or other archive formats. @@ -22,11 +22,6 @@ Metadata is included in the offers to allow the receiver to decide if they want "Offers" generally correspond to what a user might select; a single-file offer is possible but so is a directory. In both cases, they are treated as "an offer" although a directory may consist of dozens or more individual files. -Filenames are relative paths. -When sending individual files, this will be simply the filename portion (with no leading paths). -For a series of files in a directory (i.e. if a directory was selected to send) paths will be relative to that directory (starting with the directory itself). -(XXX see "file naming" in discussion) - ## Version Negotiation @@ -34,8 +29,8 @@ There is an existing file-transfer protocol which does not use Dilation (called Clients supporting newer versions of file-transfer (i.e. the one in this document) SHOULD offer backwards compatibility. In the mailbox protocol, applications can indicate version information. -The existing file-transfer protocol doesn't use this so the version information is empty (indicating "classic"). -This protocol will include a dict like: +The existing file-transfer protocol doesn't use this feature so the version information is empty (indicating "classic"). +This new protocol will include a dict like: ```json { @@ -48,32 +43,37 @@ This protocol will include a dict like: } ``` -The `version` key indicates the highest version of this Dilated File Transfer protocol that this peer understands. +The `"version"` key indicates the highest version of this Dilated File Transfer protocol that the peer understands. There is currently only one version: `1`. -The `mode` key indicates the desired mode of the peer. +The `"mode"` key indicates the desired mode of the peer. It has one of three values: * `"send"`: the peer will only send files (similar to classic transfer protocol) * `"receive"`: the peer only receive files (the flip side of the above) * `"connect"`: the peer will send and receive zero or more files before closing the session -Note that `send` and `receive` above will still use Dilation as all clients supporting this protocol must. +Note that `"send"` and `"receive"` above will still use Dilation as all clients supporting this protocol must. If a peer sends no version information at all, it will be using the classic protocol (and is thus using Transit and not Dilation for the peer-to-peer connection). -The `"features"` key is a list of message-formats / features understood by the peer. + XXX: maybe we don't strictly _need_ the mode at all? but it does keep things explicit .. which might be important for UX decisions on one or the other side? + XXX: that is: + XXX: a "receive-only" client can simply never send an offer + XXX: a "send-only" client simply (automatically) rejects any offer + +The `"features"` key points at a list of message-formats / features understood by the peer. This allows for existing messages to be extended, or for new message types to be added. Peers MUST _accept_ messages for any features they support. -Peers MUST only send messages for features in the other side's list. +Peers MUST only send messages / attributes for features in the other side's list. Only one format exists currently: `"core"`. - XXX:: maybe just lean on "version" for now? e.g. version `2` could introduce "features" + XXX:: maybe just lean on "version" for now? e.g. version `2` could introduce "features"? See "Example of Protocol Expansion" below for discussion about adding new attributes (including when we might increment the `"version"` instead of adding a new `"feature"`). The `"permission"` key specifies how to proceed with offers. Using mode `"ask"` tells the sender to pause after transmitting the first metadata message and await an answer from the peer before streaming further data. Using mode `"yes"` means the peer will accept all incoming transfers so the sender should not pause after the metadata (and instead immediately start sending data messages). -This cuts down latency for "one-way" transfers (see "Discussion") +Although a peer could implement "yes" mode by simply sending an accept message for each offer without user interaction, setting this mode cuts down latency for "one-way" transfers (see "Discussion") ## Protocol Details From dd4f11df6dd6cbf1fcb307f0d7df6a7b9f7906df Mon Sep 17 00:00:00 2001 From: meejah Date: Tue, 21 Jun 2022 16:03:32 -0600 Subject: [PATCH 16/49] dot diagram --- dilated-file-transfer.dot | 68 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 dilated-file-transfer.dot diff --git a/dilated-file-transfer.dot b/dilated-file-transfer.dot new file mode 100644 index 0000000..ed253a8 --- /dev/null +++ b/dilated-file-transfer.dot @@ -0,0 +1,68 @@ +digraph { + + start[rank=1]; + start -> connected [label="peer mode: connect \l"]; + start -> receiving [label="peer mode: send \l"]; + start -> sending [label="peer mode: receive \l"]; + + connected[rank=2]; + receiving[rank=2]; + sending[rank=2]; + + connected -> offer_start [label="subchannel open \l"]; + connected -> send_offer [label="make:Offer\lopen subchannel"]; + + receiving -> offer_start [label="subchannel open \l"]; + sending -> send_offer [label="make:Offer\lopen subchannel"]; + + subgraph cluster_receive { + rank=same; + label="incoming offer"; + color=lightseagreen; + style=filled; + node[color=white; style=filled;]; + + offer_start[color=white style=filled label="subchannel open"]; + offer_start -> got_offer [label="recv: Offer\lask: accept?"]; + got_offer -> accepting [label="answer:yes\lsend: AcceptOffer"]; + got_offer -> rejecting [label="answer:yes\lsend: RejectOffer"]; + + accepting -> accepting [label="recv: data \lwrite"]; + + recv_closed [label="closed"]; + accepting -> recv_closed [label="subchannel close \lclose file"]; + + rejecting -> recv_closed [label="subchannel close \l"] + } + + subgraph cluster_send { + rank=same; + label="outgoing offer"; + color=lightblue; + style=filled; + node[color=white; style=filled;]; + + send_offer[color=white style=filled]; + send_offer -> permission [label="send:Offer\lmode:ask"]; + permission -> stream_data [label="recv:OfferAccept\l"]; + permission -> send_closing [label="recv:OfferReject\lclose subchannel"]; + send_offer -> stream_data [label="send:Offer\lmode:yes"]; + + stream_data -> stream_data [label="have data\lsend:Data"]; + + stream_data -> send_closing [label="completed\lclose subchannel"]; + send_closed [label="closed"]; + send_closing -> send_closed [labe="subchannel close"]; + } + + // "we" close down first + receiving -> closing [label="done\lclose dilation"]; + sending -> closing [label="done\lclose dilation"]; + connected -> closing [label="done\lclose dilation"]; + closing -> done [label="dilation closed\l"] + + // "they" close down first + receiving -> done [label="dilation closed\l"]; + sending -> done [label="dilation closed\l"]; + connected -> done [label="dilation closed\l"]; +} \ No newline at end of file From 8b3ffa7b3c174411717d4b4a2069733e80191ad4 Mon Sep 17 00:00:00 2001 From: meejah Date: Tue, 21 Jun 2022 16:07:50 -0600 Subject: [PATCH 17/49] tweak diagram --- dilated-file-transfer.dot | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/dilated-file-transfer.dot b/dilated-file-transfer.dot index ed253a8..c216d80 100644 --- a/dilated-file-transfer.dot +++ b/dilated-file-transfer.dot @@ -1,14 +1,11 @@ digraph { - start[rank=1]; + start[color=blue style=bold]; + done[color=red style=bold]; start -> connected [label="peer mode: connect \l"]; start -> receiving [label="peer mode: send \l"]; start -> sending [label="peer mode: receive \l"]; - connected[rank=2]; - receiving[rank=2]; - sending[rank=2]; - connected -> offer_start [label="subchannel open \l"]; connected -> send_offer [label="make:Offer\lopen subchannel"]; @@ -16,16 +13,16 @@ digraph { sending -> send_offer [label="make:Offer\lopen subchannel"]; subgraph cluster_receive { - rank=same; label="incoming offer"; color=lightseagreen; style=filled; node[color=white; style=filled;]; offer_start[color=white style=filled label="subchannel open"]; - offer_start -> got_offer [label="recv: Offer\lask: accept?"]; + offer_start -> accepting [label="recv:Offer\lour-mode: yes"]; + offer_start -> got_offer [label="recv: Offer\lask:accept?"]; got_offer -> accepting [label="answer:yes\lsend: AcceptOffer"]; - got_offer -> rejecting [label="answer:yes\lsend: RejectOffer"]; + got_offer -> rejecting [label="answer:no\lsend: RejectOffer"]; accepting -> accepting [label="recv: data \lwrite"]; @@ -36,7 +33,6 @@ digraph { } subgraph cluster_send { - rank=same; label="outgoing offer"; color=lightblue; style=filled; From 59ade4d4ea9bcde16beaf3ad30fd3a0d4b720182 Mon Sep 17 00:00:00 2001 From: meejah Date: Wed, 22 Jun 2022 18:31:44 -0600 Subject: [PATCH 18/49] editing tweaks --- dilated-file-transfer.md | 52 ++++++++++++++++++++++++++++------------ 1 file changed, 37 insertions(+), 15 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 72860be..a6008fe 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -6,16 +6,18 @@ Both sides must support and use Dilation (see `dilation-protocol.md`). Any all-caps words ("MAY", "MUST", etc) follow RFC2119 conventions. - NOTE: there are several open questions / discussion points, some with corresponding "XXX" comments inline. + NOTE: there are several open questions / discussion points, some with corresponding "XXX" comments inline. See also "Discussion and Open Questions" ## Overview and Features -We describe a flexible, "session"-based approach to file transfer allowing either side to offer files to send while the other side may accept or reject each offer. -Either side MAY terminate the transfer session (by closing the wormhole) +This specification is an application-level Magic Wormhole protocol defining a flexible, "session"-based approach to file transfer. +Dilation must be supported by both clients (see "dilation-protocol.md"). +Client implementations can allow either side to offer files/directories to send while the other side may accept or reject each offer. +Either side MAY terminate the transfer session (by closing the wormhole). Either side MAY select a one-way mode, similar to the classic protocol. -Files are offered and sent individually, with no dependency on zip or other archive formats. +Files or directories are offered and sent individually, with no dependency on zip or other archive formats. Metadata is included in the offers to allow the receiver to decide if they want that file before the transfer begins. @@ -29,7 +31,7 @@ There is an existing file-transfer protocol which does not use Dilation (called Clients supporting newer versions of file-transfer (i.e. the one in this document) SHOULD offer backwards compatibility. In the mailbox protocol, applications can indicate version information. -The existing file-transfer protocol doesn't use this feature so the version information is empty (indicating "classic"). +The classic file-transfer protocol doesn't use this feature so the version information is empty. This new protocol will include a dict like: ```json @@ -46,6 +48,8 @@ This new protocol will include a dict like: The `"version"` key indicates the highest version of this Dilated File Transfer protocol that the peer understands. There is currently only one version: `1`. + XXX: Brian Warner also notes that a list-of-versions may be a good/better approach (even if the result "a single version"); see the Dilation version-negotiation. + The `"mode"` key indicates the desired mode of the peer. It has one of three values: * `"send"`: the peer will only send files (similar to classic transfer protocol) @@ -60,9 +64,15 @@ If a peer sends no version information at all, it will be using the classic prot XXX: a "receive-only" client can simply never send an offer XXX: a "send-only" client simply (automatically) rejects any offer + XXX: "being explicit" has the advantage that a "receive-only" peer that connects to another "receive-only" peer will discover that quickly, and can (properly) fail -- otherwise, they'll only notice when one side gets bored and quits + + XXX: two "send-only" peers that connect will fail fairly quickly -- each side should disconnect with a protocol-error when their receive the first offer + + XXX: a "connect" peer that contacts either a "receive-only" or "send-only" can benefit from "being explicit" by disabling parts of the UI (for example) that won't work (e.g. contacting a "send-only" peer means the user can't browse/drop files) + The `"features"` key points at a list of message-formats / features understood by the peer. This allows for existing messages to be extended, or for new message types to be added. -Peers MUST _accept_ messages for any features they support. +Peers MUST _accept_ messages for any features they declare in `"features"`. Peers MUST only send messages / attributes for features in the other side's list. Only one format exists currently: `"core"`. @@ -86,12 +96,14 @@ All control-channel messages are encoded using `msgpack`. --> XXX: see "message encoding" discussion Control-channel message formats are described using Python pseudo-code to illustrate the data types involved. +They are actually an encoded `Map` with `String` keys (to use `msgpack` lingo) and values as per the pseudo-code. -All control-channel messages contain an integer "kind" field describing the sort of message it is. +All control-channel messages contain a integer "kind" field describing the sort of message it is. +(That is, `"kind": 1` for example, not the single-byte tag used for subchannel messages) Rejected idea: Version message, because we already do version negotiation via mailbox features. -Rejected idea: Offer/Answer messages via the control channel: we need to open a subchannel anyway; the subchannel-IDs are not intended to be part of the Dilation API. +Rejected idea: Offer/Answer messages via the control channel: we need to open a subchannel anyway and the subchannel-IDs are not intended to be part of the Dilation public API. ### Control Channel Messages @@ -114,20 +126,21 @@ If the other peer specified `"mode": "send"` then this peer MUST NOT make any Of To make an Offer the peer opens a subchannel. Recall from the Dilation specification that subchannels are _record_ pipes (not simple byte-streams). -All records on the subchannel are bytes, where the first byte indicates the kind of message and the remaining bytes are a kind-dependant payload. +All records on the subchannel begin with a single byte indicating the kind of message. +Any additional bytes are a kind-dependant payload. The following kinds of messages exist (as indicated by the first byte): * 1: msgpack-encoded `FileOffer` message * 2: msgpack-encoded `DirectoryOffer` message * 3: msgpack-encoded `OfferAccept` message * 4: msgpack-encoded `OfferReject` message -* 5: file data bytes +* 5: raw file data bytes All other byte values are reserved for future use and MUST NOT be used. XXX: maybe spec [0, 128) as reserved, and [128, 255) for "experiments"? -The first message sent on the new subchannel is one of two sorts of offer messages. +The sender that opened the new subchannel MUST immediately send one of the two kinds of offer messages. To offer a single file (with message kind `1`): @@ -151,8 +164,7 @@ class DirectoryOffer: The filenames in the `"files"` list are sequences of unicode path-names and are relative to the `"base"` from the `DirectoryOffer` (but NOT including that part). Note that a `FileOffer` message also preceeds each file in the Directory when the data is streamed. The files MUST be streamed in the same order they appear in the `files` list. -The last segment of each entry in the filename list MUST match the `.filename` of the `FileOffer` message. - +The last segment of each entry in the filename list MUST match the `"filename"` of the `FileOffer` message. For example: @@ -198,14 +210,24 @@ When completed, the subchannel is closed. That is, the offering side always initiates the open and close of the corresponding subchannel. +For the example above, the sending side determins whether it should stream data (e.g. in mode "yes" it should start right away, otherwise wait for an `OfferAccept`). + +It will then send messages in this order: +* a kind `1` `FileOffer(filename="README")` +* a kind `5` data with 65 bytes of payload +* a kind `1` `FileOffer(filename="hellow.py")` +* a kind `5` data with 100 bytes of payload +* close the subchannel + Messages of kind `5` ("file data bytes") consist solely of file data. -A single data message MUST NOT exceed 65536 (65KiB) inculding the single byte for "kind". +A single data message MUST NOT exceed 65536 bytes (65KiB) inculding the single byte for "kind" (so 65535 maximum payload bytes). Applications are free to choose how to fragment the file data so long as no single message is bigger than 65536 bytes. A good default to choose in 2022 is 16KiB (2^14 - 1 payload bytes) XXX: what is a good default? Dilation doesn't give guidance either... -When sending a `DirectoryOffer` each individual file is preceeded by a `FileOffer` message. However the rules about "maybe wait for reply" no longer exist; that is, all file data SHOULD be immediately sent. +When sending a `DirectoryOffer` each individual file is preceeded by a `FileOffer` message. +However the rules about "maybe wait for reply" no longer exist; that is, all file data MUST be immediately sent (the `FileOffer`s serve as a header). See examples down below, after "Discussion". From d90275c4b5a4eb93dd47e179bfc3e7f7d9f71edf Mon Sep 17 00:00:00 2001 From: meejah Date: Wed, 22 Jun 2022 18:40:28 -0600 Subject: [PATCH 19/49] add philosophy --- dilated-file-transfer.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index a6008fe..19ad35e 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -9,6 +9,16 @@ Any all-caps words ("MAY", "MUST", etc) follow RFC2119 conventions. NOTE: there are several open questions / discussion points, some with corresponding "XXX" comments inline. See also "Discussion and Open Questions" +## Philosophic Note + +The place to be opinionated is in client implementations. +The place to be flexible is in protocol specifications. + +Everyone reading this likely has a particular sort of user experience (UX) in mind; the protocol should _allow_ reasonable features but shouldn't _demand_ any particular UX. + +The protocol absolutely MUST be extensible in the future (we can't do everything right now). + + ## Overview and Features This specification is an application-level Magic Wormhole protocol defining a flexible, "session"-based approach to file transfer. @@ -265,7 +275,7 @@ preliminary conclusion: msgpack. * file naming -Sending a single file like `/home/meejah/Documents/Letter.docx` gets a filename like `Letter.docx` +Sending a single file like `/home/meejah/Documents/Letter.docx` gets a filename `Letter.docx` Sending a whole directory like `/home/meejah/Documents/` would result in a directory-offer with basedir `Documents` and some number of files (possibly with sub-paths). This does NOT offer a client the chance to select "this" and "that" from a Directory offer (however, see the "Protocol Expansion Execrises" section). @@ -278,7 +288,8 @@ Preliminary conclusion: centering around "the thing a human would select" (i.e. There is no "finished" message. Maybe there should be? (e.g. the receiving side sends back a hash of the file to confirm it received it properly?) Does "re-using" the `FileOffer` as a kind of "header" when streaming `DirectoryOffer` contents make sense? -We need _something_ to indicate the next file etc +We need _something_ to indicate the next file etc. +Preliminary conclusion: it's fine and gives consistent metadata Do the limits on message size make sense? Should "65KiB" be much smaller, potentially? (Given that network conditions etc vary a lot, I think it makes sense for the _spec_ to be somewhat flexible here and "65k" doesn't seem very onerous for most modern devices / computers) From d15f61adabefb0a7aa2e7c3e6e3e900176f37804 Mon Sep 17 00:00:00 2001 From: meejah Date: Wed, 22 Jun 2022 18:46:18 -0600 Subject: [PATCH 20/49] grammar is fun --- dilated-file-transfer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 19ad35e..f041461 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -16,7 +16,7 @@ The place to be flexible is in protocol specifications. Everyone reading this likely has a particular sort of user experience (UX) in mind; the protocol should _allow_ reasonable features but shouldn't _demand_ any particular UX. -The protocol absolutely MUST be extensible in the future (we can't do everything right now). +The protocol absolutely MUST be extensible in the future (we can't do everything right, right now). ## Overview and Features From 6f545bb36cd4082c4374c5031b4405c135b95dfe Mon Sep 17 00:00:00 2001 From: meejah Date: Wed, 22 Jun 2022 18:53:33 -0600 Subject: [PATCH 21/49] links --- dilated-file-transfer.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index f041461..de5c9ad 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -6,7 +6,7 @@ Both sides must support and use Dilation (see `dilation-protocol.md`). Any all-caps words ("MAY", "MUST", etc) follow RFC2119 conventions. - NOTE: there are several open questions / discussion points, some with corresponding "XXX" comments inline. See also "Discussion and Open Questions" + NOTE: there are several open questions / discussion points, some with corresponding "XXX" comments inline. See also [Discussion and Open Questions](#discussion) ## Philosophic Note @@ -22,7 +22,7 @@ The protocol absolutely MUST be extensible in the future (we can't do everything ## Overview and Features This specification is an application-level Magic Wormhole protocol defining a flexible, "session"-based approach to file transfer. -Dilation must be supported by both clients (see "dilation-protocol.md"). +Dilation must be supported by both clients (see :ref:`dilation-protocol.md`). Client implementations can allow either side to offer files/directories to send while the other side may accept or reject each offer. Either side MAY terminate the transfer session (by closing the wormhole). Either side MAY select a one-way mode, similar to the classic protocol. @@ -93,7 +93,7 @@ See "Example of Protocol Expansion" below for discussion about adding new attrib The `"permission"` key specifies how to proceed with offers. Using mode `"ask"` tells the sender to pause after transmitting the first metadata message and await an answer from the peer before streaming further data. Using mode `"yes"` means the peer will accept all incoming transfers so the sender should not pause after the metadata (and instead immediately start sending data messages). -Although a peer could implement "yes" mode by simply sending an accept message for each offer without user interaction, setting this mode cuts down latency for "one-way" transfers (see "Discussion") +Although a peer could implement "yes" mode by simply sending an accept message for each offer without user interaction, setting this mode cuts down latency for "one-way" transfers (see [Discussion](#discussion)) ## Protocol Details @@ -242,7 +242,7 @@ However the rules about "maybe wait for reply" no longer exist; that is, all fil See examples down below, after "Discussion". -## Discussion and Open Questions +## Discussion and Open Questions {#discussion} * Overall versioning considerations From 0ba5c9acf5bf480cbe2e1a34c313b8f338068e2a Mon Sep 17 00:00:00 2001 From: meejah Date: Thu, 23 Jun 2022 21:45:29 -0600 Subject: [PATCH 22/49] old --- file-transfer-v2-complex.seqdiag | 32 -------------------------------- file-transfer-v2-simple.seqdiag | 15 --------------- 2 files changed, 47 deletions(-) delete mode 100644 file-transfer-v2-complex.seqdiag delete mode 100644 file-transfer-v2-simple.seqdiag diff --git a/file-transfer-v2-complex.seqdiag b/file-transfer-v2-complex.seqdiag deleted file mode 100644 index 1155149..0000000 --- a/file-transfer-v2-complex.seqdiag +++ /dev/null @@ -1,32 +0,0 @@ -seqdiag { - Alice -> Bian [label="OPEN(subchannel=1)"] - Alice -> Bian [label="control \n Offer[filename=foo, subchannel=1, id=42]"] - Alice -> Bian [label="OPEN(subchannel=3)"] - Alice -> Bian [label="control \n Offer[filename=bar, subchannel=3, id=89]"] - - Alice <- Bian [label="control \n Accept[id=42]"] - Alice -> Bian [label="subchannel 1 \n DATA"] - - Alice -> Bian [label="OPEN(subchannel=2)"] - Alice <- Bian [label="control \n Offer[filename=quux, subchannel=2, id=65]"] - - Alice <- Bian [label="control \n Accept[id=89]"] - Alice -> Bian [label="subchannel 3 \n DATA"] - Alice -> Bian [label="subchannel 1 \n DATA"] - Alice -> Bian [label="subchannel 3 \n DATA"] - Alice -> Bian [label="subchannel 1 \n DATA"] - - Alice -> Bian [label="control \n Accept[id=65]"] - Alice <- Bian [label="subchannel 2 \n DATA"] - - Alice -> Bian [label="CLOSE(subchannel=1)"] - - Alice <- Bian [label="subchannel 2 \n DATA"] - Alice -> Bian [label="subchannel 3 \n DATA"] - Alice -> Bian [label="subchannel 3 \n DATA"] - - Alice <- Bian [label="CLOSE(subchannel=2)"] - Alice -> Bian [label="CLOSE(subchannel=3)"] - - Alice -> Bian [label="close mailbox"] -} diff --git a/file-transfer-v2-simple.seqdiag b/file-transfer-v2-simple.seqdiag deleted file mode 100644 index 8a50b67..0000000 --- a/file-transfer-v2-simple.seqdiag +++ /dev/null @@ -1,15 +0,0 @@ -seqdiag { - Alice -> Bian [label="OPEN(subchannel=1)"] - Alice -> Bian [label="control \n Offer[filename=foo, subchannel=1, id=42]"] - - Alice <- Bian [label="control \n Accept[id=42]"] - - Alice -> Bian [label="subchannel 1 \n DATA"] - Alice -> Bian [label="subchannel 1 \n DATA"] - Alice -> Bian [label="subchannel 1 \n DATA"] - - Alice -> Bian [label="CLOSE(subchannel=1)"] - - Alice <- Bian [label="close mailbox"] - Alice -> Bian [label="close mailbox"] -} From 0ffed00a3cb7c88b224a36fb8ae602bc4c6a3c52 Mon Sep 17 00:00:00 2001 From: meejah Date: Tue, 28 Jun 2022 16:08:01 -0600 Subject: [PATCH 23/49] slight re-wording and spelling --- dilated-file-transfer.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index de5c9ad..a3cee9c 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -16,7 +16,7 @@ The place to be flexible is in protocol specifications. Everyone reading this likely has a particular sort of user experience (UX) in mind; the protocol should _allow_ reasonable features but shouldn't _demand_ any particular UX. -The protocol absolutely MUST be extensible in the future (we can't do everything right, right now). +The protocol absolutely MUST be extensible in the future (we can't do everything correctly immediately). ## Overview and Features @@ -137,7 +137,7 @@ To make an Offer the peer opens a subchannel. Recall from the Dilation specification that subchannels are _record_ pipes (not simple byte-streams). All records on the subchannel begin with a single byte indicating the kind of message. -Any additional bytes are a kind-dependant payload. +Any additional bytes are a kind-dependent payload. The following kinds of messages exist (as indicated by the first byte): * 1: msgpack-encoded `FileOffer` message @@ -172,7 +172,7 @@ class DirectoryOffer: ``` The filenames in the `"files"` list are sequences of unicode path-names and are relative to the `"base"` from the `DirectoryOffer` (but NOT including that part). -Note that a `FileOffer` message also preceeds each file in the Directory when the data is streamed. +Note that a `FileOffer` message also precedes each file in the Directory when the data is streamed. The files MUST be streamed in the same order they appear in the `files` list. The last segment of each entry in the filename list MUST match the `"filename"` of the `FileOffer` message. @@ -220,23 +220,23 @@ When completed, the subchannel is closed. That is, the offering side always initiates the open and close of the corresponding subchannel. -For the example above, the sending side determins whether it should stream data (e.g. in mode "yes" it should start right away, otherwise wait for an `OfferAccept`). +For the example above, the sending side determines whether it should stream data (e.g. in mode "yes" it should start right away, otherwise wait for an `OfferAccept`). It will then send messages in this order: * a kind `1` `FileOffer(filename="README")` * a kind `5` data with 65 bytes of payload -* a kind `1` `FileOffer(filename="hellow.py")` +* a kind `1` `FileOffer(filename="hello.py")` * a kind `5` data with 100 bytes of payload * close the subchannel Messages of kind `5` ("file data bytes") consist solely of file data. -A single data message MUST NOT exceed 65536 bytes (65KiB) inculding the single byte for "kind" (so 65535 maximum payload bytes). +A single data message MUST NOT exceed 65536 bytes (65KiB) including the single byte for "kind" (so 65535 maximum payload bytes). Applications are free to choose how to fragment the file data so long as no single message is bigger than 65536 bytes. A good default to choose in 2022 is 16KiB (2^14 - 1 payload bytes) XXX: what is a good default? Dilation doesn't give guidance either... -When sending a `DirectoryOffer` each individual file is preceeded by a `FileOffer` message. +When sending a `DirectoryOffer` each individual file is preceded by a `FileOffer` message. However the rules about "maybe wait for reply" no longer exist; that is, all file data MUST be immediately sent (the `FileOffer`s serve as a header). See examples down below, after "Discussion". @@ -258,7 +258,7 @@ A file-transfer peer supporting Dilation and this new protocol sends `"transfer" We include a `"version"` key in that dict so that this transfer protocol may be extended in the future (see "Protocol Expansion Exercises" below). Additionally, a `"features"` key is included in that information. -Although releated, this is somewhat orthogonal to "versions". +Although related, this is somewhat orthogonal to "versions". That is, a peer may _know how to parse_ some (newer) version of this protocol but may still wish to _not_ support (or use) a particular feature. @@ -268,7 +268,7 @@ While `msgpack` is mentioned above, there are several other binary-supporting li These are (in no particular order) at least: CBOR or flatbuffers or protocolbuffers or cap'n'proto We could also still decide to simply use JSON, as that is also unambiguous. -Although it enjoys widespread support, JSON suffers from a lack of 64-bit integers (because JavaScript only supports 2^53 integers) and doesn't support a byte-string type (forcing e.g. any binary data to be encded in base64 or similar). +Although it enjoys widespread support, JSON suffers from a lack of 64-bit integers (because JavaScript only supports 2^53 integers) and doesn't support a byte-string type (forcing e.g. any binary data to be encoded in base64 or similar). preliminary conclusion: msgpack. @@ -278,7 +278,7 @@ preliminary conclusion: msgpack. Sending a single file like `/home/meejah/Documents/Letter.docx` gets a filename `Letter.docx` Sending a whole directory like `/home/meejah/Documents/` would result in a directory-offer with basedir `Documents` and some number of files (possibly with sub-paths). -This does NOT offer a client the chance to select "this" and "that" from a Directory offer (however, see the "Protocol Expansion Execrises" section). +This does NOT offer a client the chance to select "this" and "that" from a Directory offer (however, see the "Protocol Expansion Exercises" section). Preliminary conclusion: centering around "the thing a human would select" (i.e. "a file" or "a directory") makes the most sense. @@ -365,7 +365,7 @@ class FileOffer: bytes: int thumbnail: bytes # introduced in "thumbnail" feature; PNG data ``` -A new peer speaking to an old peer will never see `thumbnail` in the Offers, because the old peer sent `"formats": ["core"]` so the new peer knows not to inclue that attribute (and the old peer won't ever send it). +A new peer speaking to an old peer will never see `thumbnail` in the Offers, because the old peer sent `"formats": ["core"]` so the new peer knows not to include that attribute (and the old peer won't ever send it). Two new peers speaking will both send `"formats": ["core", "thumbnails"]` and so will both include (and know how to interpret) `"thumbnail"` attributes on `Offers`. @@ -423,7 +423,7 @@ This program prints a code and waits for protocol setup. Alice uses a GUI program on `laptop` that allows drag-and-drop sending of files. In this program, she enters the code that `desktop` printed out. -Once the Dilated connection is establed, Alice can drop any number of files into the GUI application and they will be immediately written on the `desktop` without interaction. +Once the Dilated connection is established, Alice can drop any number of files into the GUI application and they will be immediately written on the `desktop` without interaction. Speaking this protocol, the `desktop` (receive-only CLI) peer sends version information: @@ -452,7 +452,7 @@ When a subchannel opens, it: * finds a `FileOffer` and opens a local file for writing * reads subsequent data records, writing them to the open file * notices the subchannel close -* double-checks that the corrent number of payload bytes were received +* double-checks that the correct number of payload bytes were received * XXX: return a checksum ack? (this might be more in line with Waterken principals so the sending side knows to delete state relating to this file ... but arguably with Dilation it "knows" that file made it?) * closes the file From 83b8dae25ca1892532f9c183e06f919c1edf5231 Mon Sep 17 00:00:00 2001 From: meejah Date: Thu, 30 Jun 2022 15:47:27 -0600 Subject: [PATCH 24/49] clarify --- dilated-file-transfer.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index a3cee9c..da61c52 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -193,9 +193,9 @@ This indicates an offer to send a directory consisting of two files: one in `"pr What happens next depends on the mode of the peer. -If the peer has `"mode": "yes"` then this peer MUST immediately start sending content messages (see below). +If the other peer has `"mode": "yes"` then this peer MUST immediately start sending content messages (see below). -If the peer has `"mode": "ask"` then this peer MUST NOT send any more messages and instead await an incoming message. +If the other peer has `"mode": "ask"` then this peer MUST NOT send any more messages and instead await an incoming message. That incoming message MUST be one of two reply messages: `OfferAccept` or `OfferReject`. These are indicated by the kind byte of that message being `3` or `4` (see list above). From b650bf8460ce92b6bb807494c80dfa03785509d6 Mon Sep 17 00:00:00 2001 From: meejah Date: Thu, 30 Jun 2022 15:50:42 -0600 Subject: [PATCH 25/49] old --- file-transfer-v2.dot | 92 -------------------------------------------- 1 file changed, 92 deletions(-) delete mode 100644 file-transfer-v2.dot diff --git a/file-transfer-v2.dot b/file-transfer-v2.dot deleted file mode 100644 index 61dc9c7..0000000 --- a/file-transfer-v2.dot +++ /dev/null @@ -1,92 +0,0 @@ -digraph { - label="File Transfer v2 state-machine"; - labelfontsize=40; - labelfontname="Source Code Pro"; - pack=true; - rankdir="TB"; - ranksep="1.0 equally"; - nodesep=1.5; -/*graph [nodesep=1.5];*/ -node[]; -edge [labelfloat=true,labelfontsize=16,]; - - await_dilation [style=bold,fontcolor=blue]; - - // user creates an Offer (before connection has completed) - await_dilation -> create_offer0[arrowhead=none]; - create_offer0 -> await_dilation; - create_offer0[shape=box,color=sienna,label="create:Offer\lqueue_offer()"]; - - // the Dilation connection becomes available - await_dilation -> got_connect[arrowhead=none]; - got_connect -> connected; - got_connect[shape=box,color=sienna,label="connected\lsend_queued_offers()"] - - // other side sends offer, we notify our human - connected -> got_offer[arrowhead=none]; - got_offer -> connected; - got_offer[shape=box,color=purple,label="got:Offer\lnotify:offer_received"]; - - // other side rejects our offer, notify human - connected -> reject_offer[arrowhead=none]; - reject_offer -> connected; - reject_offer[shape=box,color=purple,label="got:Reject\lnotify:offer_rejected"]; - - // other side accepts our offer, send file - connected -> got_accept[arrowhead=none]; - got_accept -> connected; - got_accept[shape=box,color=purple,label="got:Accept\lsend_file()"]; - - // human tells us to stop, shut down - connected -> send_stop[arrowhead=none]; - send_stop -> closing; - send_stop[shape=box,color=sienna,label="stop\lclose_mailbox()"]; - - // user creates an Offer (while connected) - connected -> create_offer2[arrowhead=none]; - create_offer2 -> connected; - create_offer2[shape=box,color=sienna,label="create:Offer\lsend_offer()"]; - - // our human accepts an offer, download the file - connected -> accept_offer[arrowhead=none]; - accept_offer -> connected; - accept_offer[shape=box,color=sienna,label="accept Offer\lreceive_file()"]; - - // mailbox confirms close - closing -> await_close[arrowhead=none]; - await_close -> done; - await_close[shape=box,color=purple,label="mailbox_closed\lnotify:finished"]; - - done [style=bold,fontcolor=blue]; - - - // bunch of error cases .. technically there's probably a few - // more (like if we receive anything at all in "closing" or - // "closing_error") - // - // uncomment to .. clutter the diagram - - /* - - // get Offer before Version - await_version -> offer_err0[arrowhead=none,color=red]; - offer_err0 -> closing_error[color=red]; - offer_err0[shape=box,color=purple,label="got:Offer\lclose_mailbox()"] - - // get Accept before Version - await_version -> offer_ans0[arrowhead=none,color=red]; - offer_ans0 -> closing_error[color=red]; - offer_ans0[shape=box,color=purple,label="got:Accept\lclose_mailbox()"] - - // get Reject before Version - await_version -> offer_rej0[arrowhead=none,color=red]; - offer_rej0 -> closing_error[color=red]; - offer_rej0[shape=box,color=purple,label="got:Reject\lclose_mailbox()"] - - // closing_error worked - closing_error -> err_close[arrowhead=none,color=red]; - err_close -> done[color=red]; - err_close[shape=box,color=purple,label="mailbox_closed\lnotify:finished_error"]; - - */ -} \ No newline at end of file From 63ed3b72c04a7d4271879f68957b6248b46bb742 Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 4 Jul 2022 12:19:38 -0600 Subject: [PATCH 26/49] q: compression? --- dilated-file-transfer.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index da61c52..7a09e86 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -34,6 +34,8 @@ Metadata is included in the offers to allow the receiver to decide if they want "Offers" generally correspond to what a user might select; a single-file offer is possible but so is a directory. In both cases, they are treated as "an offer" although a directory may consist of dozens or more individual files. +XXX include compression in this revision, or allow that to be a future enhancement? + ## Version Negotiation From aaf49b029f78006499c28bded6028d8b26290c86 Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 4 Jul 2022 12:21:55 -0600 Subject: [PATCH 27/49] subtly change version declartions; add more version discussion / examples --- dilated-file-transfer.md | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 7a09e86..433890b 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -48,19 +48,32 @@ This new protocol will include a dict like: ```json { - "transfer": { - "version": 1, + "transfer-v1": { "mode": "{send|receive|connect}", - "features": ["core"], - "permission": "{ask|yes}" + "features": [], } } ``` -The `"version"` key indicates the highest version of this Dilated File Transfer protocol that the peer understands. +The version of the protocol is the `"-v1"` tag in `"transfer-v1"`. +A peer supporting newer versions may include `"transfer-v2"` or `"transfer-v3"` etc. There is currently only one version: `1`. +Versions are considered as integers, so the version tag MUST always be the entire tail of the string, MUST start with `-v` and MUST end ONLY with digits (that are an integer version bigger than 0). + +When multiple versions are present, a peer decides which version to use by comparing the "list of versions" that they each support and selects the highest from the intersection of these. +For example, if 3 versions existed, the two peers may present their version information like: + +``` + peer A: "transfer-v1": {}, "transfer-v3": {} + peer B: "transfer-v1": {}, "transfer-v2": {} +``` + +Each peer makes a list of versions the other peer accepts: `A=[1, 3]` and `B=[1, 2]`. +Taking the intersection of these yields the list `[1]` and the biggest number in that list is "1" so that is the version selected. +When possible, peers SHOULD provide backwards compatibility. +Note that you must declare each previous version supported (this allows support for any older version to be withdrawn by implementations). - XXX: Brian Warner also notes that a list-of-versions may be a good/better approach (even if the result "a single version"); see the Dilation version-negotiation. +### `"transfer-v1"`: The `"mode"` key indicates the desired mode of the peer. It has one of three values: From 63b41b169979259e5218341639428781016f5146 Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 4 Jul 2022 12:23:11 -0600 Subject: [PATCH 28/49] decide: yes, we want send/receive modes --- dilated-file-transfer.md | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 433890b..07e13ad 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -81,19 +81,12 @@ It has one of three values: * `"receive"`: the peer only receive files (the flip side of the above) * `"connect"`: the peer will send and receive zero or more files before closing the session -Note that `"send"` and `"receive"` above will still use Dilation as all clients supporting this protocol must. -If a peer sends no version information at all, it will be using the classic protocol (and is thus using Transit and not Dilation for the peer-to-peer connection). - - XXX: maybe we don't strictly _need_ the mode at all? but it does keep things explicit .. which might be important for UX decisions on one or the other side? - XXX: that is: - XXX: a "receive-only" client can simply never send an offer - XXX: a "send-only" client simply (automatically) rejects any offer - - XXX: "being explicit" has the advantage that a "receive-only" peer that connects to another "receive-only" peer will discover that quickly, and can (properly) fail -- otherwise, they'll only notice when one side gets bored and quits +If both peers indicate `"receive"` then nothing will ever happen so they both SHOULD end the session and disconnect. +If both peers indicate `"send"` then they SHOULD also end the session (although whichever sends the first Offer will induce a protocol error in the other peer). +If one peer indicates `"connect"` and the other indicates either `"send"` or `"receive"` then the peers can still interoperate and the `"connect"` side MUST continue (although it MAY indicate the peer's lack of one capability e.g. by disabling part of its UI). - XXX: two "send-only" peers that connect will fail fairly quickly -- each side should disconnect with a protocol-error when their receive the first offer - - XXX: a "connect" peer that contacts either a "receive-only" or "send-only" can benefit from "being explicit" by disabling parts of the UI (for example) that won't work (e.g. contacting a "send-only" peer means the user can't browse/drop files) +Note that `"send"` and `"receive"` modes will still use Dilation as all clients supporting this protocol must. +If a peer sends no version information at all, it will be using the classic protocol (and is thus using Transit and not Dilation for the peer-to-peer connection). The `"features"` key points at a list of message-formats / features understood by the peer. This allows for existing messages to be extended, or for new message types to be added. From 8a57ad049d3695913bc779ccfa4a683f8d14934e Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 4 Jul 2022 12:25:11 -0600 Subject: [PATCH 29/49] use strings for kind in control messages --- dilated-file-transfer.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 07e13ad..d19eedd 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -117,7 +117,7 @@ Control-channel message formats are described using Python pseudo-code to illust They are actually an encoded `Map` with `String` keys (to use `msgpack` lingo) and values as per the pseudo-code. All control-channel messages contain a integer "kind" field describing the sort of message it is. -(That is, `"kind": 1` for example, not the single-byte tag used for subchannel messages) +(That is, `"kind": "text"` for example, not the single-byte tag used for subchannel messages) Rejected idea: Version message, because we already do version negotiation via mailbox features. @@ -132,7 +132,7 @@ These messages look like: ```python class Message: message: str # unicode string - kind: int = 1 # "text message" + kind: str = "text" ``` From 442624900401951df41935fbd7195d28eabb5153 Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 4 Jul 2022 12:26:05 -0600 Subject: [PATCH 30/49] clarify further --- dilated-file-transfer.md | 1 + 1 file changed, 1 insertion(+) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index d19eedd..7761e54 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -140,6 +140,7 @@ class Message: Either side MAY propose any number of Offers at any time after the connection is set up. If the other peer specified `"mode": "send"` then this peer MUST NOT make any Offers. +If this peer specified `"mode": "receive"` then this peer MUST NOT make any Offers. To make an Offer the peer opens a subchannel. Recall from the Dilation specification that subchannels are _record_ pipes (not simple byte-streams). From 55c8eae0f3370cea76afcabca6b302590ffdbb4f Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 4 Jul 2022 12:27:14 -0600 Subject: [PATCH 31/49] tweak wording --- dilated-file-transfer.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 7761e54..facba66 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -246,7 +246,7 @@ A good default to choose in 2022 is 16KiB (2^14 - 1 payload bytes) XXX: what is a good default? Dilation doesn't give guidance either... When sending a `DirectoryOffer` each individual file is preceded by a `FileOffer` message. -However the rules about "maybe wait for reply" no longer exist; that is, all file data MUST be immediately sent (the `FileOffer`s serve as a header). +However the rules about "wait for reply" no longer exist; that is, all file data MUST be immediately sent (the `FileOffer`s serve as a header). See examples down below, after "Discussion". @@ -328,7 +328,7 @@ Given a hypothetical directory tree: * hello.py As spec'd above, if the human selects `/home/meejah/project/src/hello.py` then it should be sent as `hello.py`. -However if they select `/home/meejah/project/` then there should be a Directory Offer offers like: +However if they select `/home/meejah/project/` then there should be a Directory Offer like: ```python DirectoryOffer( From 477cf6063cbea1a2f50fd39075415b7e4442e01f Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 4 Jul 2022 12:29:19 -0600 Subject: [PATCH 32/49] get rid of 'core', no features currently --- dilated-file-transfer.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index facba66..e311c57 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -92,7 +92,8 @@ The `"features"` key points at a list of message-formats / features understood b This allows for existing messages to be extended, or for new message types to be added. Peers MUST _accept_ messages for any features they declare in `"features"`. Peers MUST only send messages / attributes for features in the other side's list. -Only one format exists currently: `"core"`. +Since there are only the core features currently, the only valid value is an empty list. +Peers MUST expect any strings in this list in the future (e.g. if a new feature is added, the protocol version isn't necessarily bumped). XXX:: maybe just lean on "version" for now? e.g. version `2` could introduce "features"? @@ -359,11 +360,11 @@ Let us suppose we decide to add `thumbnail: bytes` to the `Offer` messages. It is reasonable to imagine that some clients may not make use of this feature at all (e.g. CLI programs) and so work and bandwidth can be saved by not producing and sending them. This becomes a new `"feature"` in the protocol. -That is, the version information is upgraded to allow `"features": ["core", "thumbnails"]`. +That is, the version information is upgraded to allow `"features": ["thumbnails"]`. Peers that do not understand (or do not _want_) thumbnails do not include that in their `"features"` list. So, according to the protocol, these peers should never receive anything related to thumbnails. -Only if both peers include `"features": ["core", "thumbnails"]` will they receive thumbnail-related information. +Only if both peers include `"features": ["thumbnails"]` will they receive thumbnail-related information. The thumbnail feature itself could be implemented by expanding the `Offer` message: From 5fc4c1797e6dec64006d6fc3b20f80cfdc2f5d92 Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 4 Jul 2022 12:34:09 -0600 Subject: [PATCH 33/49] make 'permission' mode a future enhancement --- dilated-file-transfer.md | 47 ++++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 21 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index e311c57..b42b947 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -99,11 +99,6 @@ Peers MUST expect any strings in this list in the future (e.g. if a new feature See "Example of Protocol Expansion" below for discussion about adding new attributes (including when we might increment the `"version"` instead of adding a new `"feature"`). -The `"permission"` key specifies how to proceed with offers. -Using mode `"ask"` tells the sender to pause after transmitting the first metadata message and await an answer from the peer before streaming further data. -Using mode `"yes"` means the peer will accept all incoming transfers so the sender should not pause after the metadata (and instead immediately start sending data messages). -Although a peer could implement "yes" mode by simply sending an accept message for each offer without user interaction, setting this mode cuts down latency for "one-way" transfers (see [Discussion](#discussion)) - ## Protocol Details @@ -201,12 +196,7 @@ DirectoryOffer( This indicates an offer to send a directory consisting of two files: one in `"project/README"` and the other in `"project/src/hello.py"`. -What happens next depends on the mode of the peer. - -If the other peer has `"mode": "yes"` then this peer MUST immediately start sending content messages (see below). - -If the other peer has `"mode": "ask"` then this peer MUST NOT send any more messages and instead await an incoming message. - +The peer making the Offer then awaits a message from the other peer. That incoming message MUST be one of two reply messages: `OfferAccept` or `OfferReject`. These are indicated by the kind byte of that message being `3` or `4` (see list above). @@ -226,13 +216,11 @@ When the offering side gets an `OfferReject` message, the subchannel SHOULD be i The offering side MAY show the "reason" string to the user. When the offering side gets an `OfferAccept` message it begins streaming the file over the already-opened subchannel. -When completed, the subchannel is closed. +When completed, the subchannel is closed (by the peer that made the Offer). That is, the offering side always initiates the open and close of the corresponding subchannel. -For the example above, the sending side determines whether it should stream data (e.g. in mode "yes" it should start right away, otherwise wait for an `OfferAccept`). - -It will then send messages in this order: +If the receiving side responds with `OfferAccept` then (following the example above) this peer will send messages in this order: * a kind `1` `FileOffer(filename="README")` * a kind `5` data with 65 bytes of payload * a kind `1` `FileOffer(filename="hello.py")` @@ -375,13 +363,31 @@ class FileOffer: bytes: int thumbnail: bytes # introduced in "thumbnail" feature; PNG data ``` -A new peer speaking to an old peer will never see `thumbnail` in the Offers, because the old peer sent `"formats": ["core"]` so the new peer knows not to include that attribute (and the old peer won't ever send it). +A new peer speaking to an old peer will never see `thumbnail` in the Offers, because the old peer sent `"formats": []` so the new peer knows not to include that attribute (and the old peer won't ever send it). -Two new peers speaking will both send `"formats": ["core", "thumbnails"]` and so will both include (and know how to interpret) `"thumbnail"` attributes on `Offers`. +Two new peers speaking will both send `"formats": ["thumbnails"]` and so will both include (and know how to interpret) `"thumbnail"` attributes on `Offers`. Additionally, a new peer that _doesn't want_ to see `"thumbnail"` data (e.g. it's a CLI client) can simply not include `"thumbnail"` in their `"formats"` list even if their protocol implementation knows about it. +### No-Permission Mode + +An earlier draft of this included a `"permission"` key in the version information. + +Using `"permission": "yes"` tells other peer to not bother awaiting an answer to any Offers because it will accept them all (while `"permission": "ask"`, or simply nothing, selects the default behavior). + +While this _could_ be implemented by clients simply replying automatically with an OfferAccept message to all offers, having a way to select this mode allows for lower-latency (by skipping round-trips). + +This alters the behavior of both sides: the offering peer must now sometimes wait for an OfferAccept message, and sometimes simply proceed and the receiving peer either sends an OfferAccept/OfferReject or merely waits for data. + +Since there is a change to the sent `"version"` information, this needs a new protocol version. +This change also affects behavior of both peers, so it seems like that could also be a reason to upgrade the protocol version. + +So, `"transfer-v2"` would be introduced, with a new `"permsision": {"ask"|"yes"}` configuration allowed. +All other keys, abilities and features of `"transfer-v1"` would be retained in `-v2`. +A peer supporting this would then include both a `"transfer-v1"` and `"transver-v2"` key in their application versions message. + + ### Finer Grained Permissions What if we decide we want to expand the "ask" behavior to sub-items in a DirectoryOffer. @@ -442,17 +448,15 @@ Speaking this protocol, the `desktop` (receive-only CLI) peer sends version info "transfer": { "version": 1, "mode": "receive", - "features": ["core"], - "permission": "yes" + "features": [], } } ``` -The `laptop` peer then knows to not pause on its subchannels to await permission (`"permission": "yes"`). -This saves one round-trip per file. For each file that Alice drops, the `laptop` peer: * opens a subchannel * sends a `FileOffer` / kind=`1` record +* waits for the peer's answer * immediately starts sending data (via kind=`5` records) * closes the subchannel (when all data is sent) @@ -460,6 +464,7 @@ On the `desktop` peer, the program waits for subchannels to open. When a subchannel opens, it: * reads the first record * finds a `FileOffer` and opens a local file for writing +* sends an `OfferAccept` message immediately * reads subsequent data records, writing them to the open file * notices the subchannel close * double-checks that the correct number of payload bytes were received From 88b124bee67ddd3f5aa4fa500d0cfa2ca236c582 Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 4 Jul 2022 15:38:30 -0600 Subject: [PATCH 34/49] features: list -> dict --- dilated-file-transfer.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index b42b947..9e3c8e4 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -50,7 +50,7 @@ This new protocol will include a dict like: { "transfer-v1": { "mode": "{send|receive|connect}", - "features": [], + "features": {}, } } ``` @@ -88,7 +88,8 @@ If one peer indicates `"connect"` and the other indicates either `"send"` or `"r Note that `"send"` and `"receive"` modes will still use Dilation as all clients supporting this protocol must. If a peer sends no version information at all, it will be using the classic protocol (and is thus using Transit and not Dilation for the peer-to-peer connection). -The `"features"` key points at a list of message-formats / features understood by the peer. +The `"features"` key points at a dict mapping features to their configuration. +Each feature may have an arbitrary mapping of feature-specific options. This allows for existing messages to be extended, or for new message types to be added. Peers MUST _accept_ messages for any features they declare in `"features"`. Peers MUST only send messages / attributes for features in the other side's list. @@ -348,11 +349,11 @@ Let us suppose we decide to add `thumbnail: bytes` to the `Offer` messages. It is reasonable to imagine that some clients may not make use of this feature at all (e.g. CLI programs) and so work and bandwidth can be saved by not producing and sending them. This becomes a new `"feature"` in the protocol. -That is, the version information is upgraded to allow `"features": ["thumbnails"]`. +That is, the version information is upgraded to allow `"features": {"thumbnails": {}}`. Peers that do not understand (or do not _want_) thumbnails do not include that in their `"features"` list. So, according to the protocol, these peers should never receive anything related to thumbnails. -Only if both peers include `"features": ["thumbnails"]` will they receive thumbnail-related information. +Only if both peers include `"features": {"thumbnails": {}}` will they receive thumbnail-related information. The thumbnail feature itself could be implemented by expanding the `Offer` message: @@ -363,11 +364,11 @@ class FileOffer: bytes: int thumbnail: bytes # introduced in "thumbnail" feature; PNG data ``` -A new peer speaking to an old peer will never see `thumbnail` in the Offers, because the old peer sent `"formats": []` so the new peer knows not to include that attribute (and the old peer won't ever send it). +A new peer speaking to an old peer will never see `thumbnail` in the Offers, because the old peer sent `"features": {}` so the new peer knows not to include that attribute (and the old peer won't ever send it). -Two new peers speaking will both send `"formats": ["thumbnails"]` and so will both include (and know how to interpret) `"thumbnail"` attributes on `Offers`. +Two new peers speaking will both send `"features": {"thumbnails": {}}` and so will both include (and know how to interpret) `"thumbnail"` attributes on `Offers`. -Additionally, a new peer that _doesn't want_ to see `"thumbnail"` data (e.g. it's a CLI client) can simply not include `"thumbnail"` in their `"formats"` list even if their protocol implementation knows about it. +Additionally, a new peer that _doesn't want_ to see `"thumbnail"` data (e.g. it's a CLI client) can simply not include `"thumbnail"` in their `"features"` list even if their protocol implementation knows about it. ### No-Permission Mode From 0a7992aab7383b11cae5e31573d34809525828d2 Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 4 Jul 2022 15:38:37 -0600 Subject: [PATCH 35/49] reword --- dilated-file-transfer.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 9e3c8e4..f3757f3 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -375,13 +375,13 @@ Additionally, a new peer that _doesn't want_ to see `"thumbnail"` data (e.g. it' An earlier draft of this included a `"permission"` key in the version information. -Using `"permission": "yes"` tells other peer to not bother awaiting an answer to any Offers because it will accept them all (while `"permission": "ask"`, or simply nothing, selects the default behavior). +Using `"permission": "yes"` tells the other peer to not bother awaiting an answer to any Offers because it will accept them all (while `"permission": "ask"`, or simply nothing, selects the default behavior). While this _could_ be implemented by clients simply replying automatically with an OfferAccept message to all offers, having a way to select this mode allows for lower-latency (by skipping round-trips). This alters the behavior of both sides: the offering peer must now sometimes wait for an OfferAccept message, and sometimes simply proceed and the receiving peer either sends an OfferAccept/OfferReject or merely waits for data. -Since there is a change to the sent `"version"` information, this needs a new protocol version. +Since there is a change to the sent `"transfer-v1"` versioning information, this needs a new protocol version. This change also affects behavior of both peers, so it seems like that could also be a reason to upgrade the protocol version. So, `"transfer-v2"` would be introduced, with a new `"permsision": {"ask"|"yes"}` configuration allowed. From 1b91a7474f7aaed57f241b7bc2e55b684da6e01d Mon Sep 17 00:00:00 2001 From: meejah Date: Mon, 4 Jul 2022 15:39:08 -0600 Subject: [PATCH 36/49] clean up examples, expand compression --- dilated-file-transfer.md | 31 ++++++++----------------------- 1 file changed, 8 insertions(+), 23 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index f3757f3..55f6a1d 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -393,40 +393,25 @@ A peer supporting this would then include both a `"transfer-v1"` and `"transver- What if we decide we want to expand the "ask" behavior to sub-items in a DirectoryOffer. -As this affects the behavior of both the sender (who now has to wait more often) and the receiver (who now has to send a new message) this means an overall protocol version bump. +Although this affects the behavior of both the sender (who now has to wait more often) and the receiver (who now has to send a new message) we could simply use a `"features"` flag. - XXX: _does_ it though? I think we could use `"formats"` here too... - -So, this means that `"version": 2` would be the newest version. -Any peer that sends a version lower than this (i.e. the existing `1`) will not send any fine-grained information (or "yes" messages). -Any peer who sees the other side at a version lower than `2` thus cannot use this behavior and has to pretend to be a version `1` peer. - -If both peers send `2` then they can both use the new behavior (still using the overall `"yes"` versus `"ask"` switch that exists now, probably). +Note that this probably interacts with the "no-permission mode" as well; a peer supporting that would want that behavior in DirectoryOffer situations as well. ### Compression Suppose that we decide to introduce compression. -(XXX again, probably just leverage "features"?) - - -### Big Change - -What is a sort of change that might actually _require_ us to use the `"version": 2` lever? - - -### How to Represent Overall Version - -It can be useful to send a "list of versions" you support even if the ultimate outcome of a "version negotiation" is a single scalar (of "maximum version"). +Compression is introduced as a new feature-flag, `"features": {"compression-zstd": {"level": 3}}` specifying Zstandard compression. -Something to do with being able to release (and then revoke) particular (possibly "experimental") versions. +The compression-level is set to 3 by an option to this feature. -There may be an example in the TLS history surrounding this. +Every _offer_ uses a single compression context, streaming all bytes through it in sequence (even if the offer was a DirectoryOffer). +That is, in the DirectoryOffer case, every file is streamed through one compression context. +(Note that the compressor will need to `flush()` after each file to ensure all bytes are written before the next FileOffer header as sent). -This means we might want `"version": [1, 2]` for example instead of `"version": 1` or `"version": 2` alone. +Rejected idea: stream _all_ offers through one context .. but implementations may be doing one thread per subchannel or similar and contexts are not thread-safe. - XXX expand, find TLS example ## Example: one-way transfer From f61ec7a572e915579aaa4028b08835426eb6ae97 Mon Sep 17 00:00:00 2001 From: meejah Date: Tue, 5 Jul 2022 14:03:07 -0600 Subject: [PATCH 37/49] no version 1 --- dilated-file-transfer.md | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 55f6a1d..5b5dc41 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -48,32 +48,35 @@ This new protocol will include a dict like: ```json { - "transfer-v1": { + "transfer-v2": { "mode": "{send|receive|connect}", "features": {}, } } ``` -The version of the protocol is the `"-v1"` tag in `"transfer-v1"`. -A peer supporting newer versions may include `"transfer-v2"` or `"transfer-v3"` etc. -There is currently only one version: `1`. +The version of the protocol is the `"-v2"` tag in `"transfer-v2"`. +A peer supporting newer versions may include `"transfer-v3"` or `"transfer-v4"` etc. +There is currently only one version: `2`. +In order to avoid confusion with "classic" transfer, this protocol skips "-v1" -- there is no version 1, we start at 2. + Versions are considered as integers, so the version tag MUST always be the entire tail of the string, MUST start with `-v` and MUST end ONLY with digits (that are an integer version bigger than 0). When multiple versions are present, a peer decides which version to use by comparing the "list of versions" that they each support and selects the highest from the intersection of these. For example, if 3 versions existed, the two peers may present their version information like: ``` - peer A: "transfer-v1": {}, "transfer-v3": {} - peer B: "transfer-v1": {}, "transfer-v2": {} + peer A: "transfer-v2": {}, "transfer-v4": {} + peer B: "transfer-v2": {}, "transfer-v3": {} ``` -Each peer makes a list of versions the other peer accepts: `A=[1, 3]` and `B=[1, 2]`. -Taking the intersection of these yields the list `[1]` and the biggest number in that list is "1" so that is the version selected. +Each peer makes a list of versions the other peer accepts: `A=[2, 4]` and `B=[2, 3]`. +Taking the intersection of these yields the list `[2]` and the biggest number in that list is "2" so that is the version selected. When possible, peers SHOULD provide backwards compatibility. Note that you must declare each previous version supported (this allows support for any older version to be withdrawn by implementations). -### `"transfer-v1"`: + +### `"transfer-v2"`: The `"mode"` key indicates the desired mode of the peer. It has one of three values: @@ -381,12 +384,12 @@ While this _could_ be implemented by clients simply replying automatically with This alters the behavior of both sides: the offering peer must now sometimes wait for an OfferAccept message, and sometimes simply proceed and the receiving peer either sends an OfferAccept/OfferReject or merely waits for data. -Since there is a change to the sent `"transfer-v1"` versioning information, this needs a new protocol version. +Since there is a change to the sent `"transfer-v2"` versioning information, this needs a new protocol version. This change also affects behavior of both peers, so it seems like that could also be a reason to upgrade the protocol version. -So, `"transfer-v2"` would be introduced, with a new `"permsision": {"ask"|"yes"}` configuration allowed. -All other keys, abilities and features of `"transfer-v1"` would be retained in `-v2`. -A peer supporting this would then include both a `"transfer-v1"` and `"transver-v2"` key in their application versions message. +So, `"transfer-v3"` would be introduced, with a new `"permsision": {"ask"|"yes"}` configuration allowed. +All other keys, abilities and features of `"transfer-v2"` would be retained in `-v3`. +A peer supporting this would then include both a `"transfer-v2"` and `"transver-v3"` key in their application versions message. ### Finer Grained Permissions From dd244164996634aa2ceb7244ced676dc456b76f0 Mon Sep 17 00:00:00 2001 From: meejah Date: Tue, 5 Jul 2022 14:16:26 -0600 Subject: [PATCH 38/49] add a final message with checksum hash --- dilated-file-transfer.md | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 5b5dc41..5401f14 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -154,6 +154,7 @@ The following kinds of messages exist (as indicated by the first byte): * 3: msgpack-encoded `OfferAccept` message * 4: msgpack-encoded `OfferReject` message * 5: raw file data bytes +* 6: msgpack-encoded `FileAcknowledge` message All other byte values are reserved for future use and MUST NOT be used. @@ -170,6 +171,16 @@ class FileOffer: bytes: int # total number of bytes in the file ``` +At the end of the file data, _both_ sides MUST send a `FileAcknowledge` message: + +```python +class FileAcknowledge: + bytes: int # total number of bytes in the file + hash: bytes # 32-byte Blake2b hash of the file +``` + +If the hash from the other peer does not match the one this peer sent, it is an error. + To offer a directory tree of many files (with message kind `2`): ```python @@ -227,8 +238,10 @@ That is, the offering side always initiates the open and close of the correspond If the receiving side responds with `OfferAccept` then (following the example above) this peer will send messages in this order: * a kind `1` `FileOffer(filename="README")` * a kind `5` data with 65 bytes of payload +* a kind `6` `FileAcknowledge` with size and hash * a kind `1` `FileOffer(filename="hello.py")` * a kind `5` data with 100 bytes of payload +* a kind `6` `FileAcknowledge` with size and hash * close the subchannel Messages of kind `5` ("file data bytes") consist solely of file data. @@ -238,7 +251,7 @@ A good default to choose in 2022 is 16KiB (2^14 - 1 payload bytes) XXX: what is a good default? Dilation doesn't give guidance either... -When sending a `DirectoryOffer` each individual file is preceded by a `FileOffer` message. +When sending a `DirectoryOffer` each individual file is preceded by a `FileOffer` message and ends with a `FileAcknowledge` message. However the rules about "wait for reply" no longer exist; that is, all file data MUST be immediately sent (the `FileOffer`s serve as a header). See examples down below, after "Discussion". @@ -434,8 +447,7 @@ Speaking this protocol, the `desktop` (receive-only CLI) peer sends version info ```json { - "transfer": { - "version": 1, + "transfer-v2": { "mode": "receive", "features": [], } @@ -447,6 +459,8 @@ For each file that Alice drops, the `laptop` peer: * sends a `FileOffer` / kind=`1` record * waits for the peer's answer * immediately starts sending data (via kind=`5` records) +* sends a `FileAcknowledge` / kind=`6` record +* waits for a `FileAcknowledge` from the peer * closes the subchannel (when all data is sent) On the `desktop` peer, the program waits for subchannels to open. @@ -455,9 +469,10 @@ When a subchannel opens, it: * finds a `FileOffer` and opens a local file for writing * sends an `OfferAccept` message immediately * reads subsequent data records, writing them to the open file +* reads a `FileAcknowledge` / kind=`6` record from the peer +* sends a `FileAcknowledge` / kind=`6` record (note that it should generate its own hash) +* double-checks that the correct number of payload bytes were received and that the hash sent matches the hash received * notices the subchannel close -* double-checks that the correct number of payload bytes were received - * XXX: return a checksum ack? (this might be more in line with Waterken principals so the sending side knows to delete state relating to this file ... but arguably with Dilation it "knows" that file made it?) * closes the file When the GUI application finishes (e.g. Alice closes it) the mailbox is closed. From 7705f33eedd6775d5c7f130fa9a986dd8ea3016f Mon Sep 17 00:00:00 2001 From: meejah Date: Tue, 5 Jul 2022 14:24:46 -0600 Subject: [PATCH 39/49] clarify --- dilated-file-transfer.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 5401f14..a6b6cfd 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -211,6 +211,14 @@ DirectoryOffer( This indicates an offer to send a directory consisting of two files: one in `"project/README"` and the other in `"project/src/hello.py"`. +When constructing a `DirectoryOffer`, the peer will be recursively searching all files in the directory or otherwise discovering the names and sizes of files to send. +Peers MUST remember the stat information (at least size + modification-time) of each file in the offer. +When sending file data peers MUST compare the current size + modification-time to that recorded when making the offer. +Because the files actually sent MUST match the offer, any files that have appeared since the offer MUST be ignored. +If any promised files have disappeared or their modification-time or size has changed, an error MUST be signalled and the transfer aborted. + +That is, peers MUST ensure that their offer matches the files sent. + The peer making the Offer then awaits a message from the other peer. That incoming message MUST be one of two reply messages: `OfferAccept` or `OfferReject`. These are indicated by the kind byte of that message being `3` or `4` (see list above). From 7792aaf1c150ae5eb402a2e183eae3593282f28d Mon Sep 17 00:00:00 2001 From: meejah Date: Tue, 5 Jul 2022 14:32:16 -0600 Subject: [PATCH 40/49] clarify --- dilated-file-transfer.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index a6b6cfd..6ee8535 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -88,6 +88,8 @@ If both peers indicate `"receive"` then nothing will ever happen so they both SH If both peers indicate `"send"` then they SHOULD also end the session (although whichever sends the first Offer will induce a protocol error in the other peer). If one peer indicates `"connect"` and the other indicates either `"send"` or `"receive"` then the peers can still interoperate and the `"connect"` side MUST continue (although it MAY indicate the peer's lack of one capability e.g. by disabling part of its UI). +That is, existing applications expecting "one-way" transfers as with the classic protocol can use `"receive"` or `"send"` mode and later introduce the more-flexible `"connect"` mode that -- if both peers select it -- allows for multiple independant transfers in either direction. + Note that `"send"` and `"receive"` modes will still use Dilation as all clients supporting this protocol must. If a peer sends no version information at all, it will be using the classic protocol (and is thus using Transit and not Dilation for the peer-to-peer connection). From 660634775958b7588693e3d19c1b7bb681810cac Mon Sep 17 00:00:00 2001 From: meejah Date: Fri, 8 Jul 2022 01:55:04 -0600 Subject: [PATCH 41/49] add symlink discussion --- dilated-file-transfer.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 6ee8535..900d99d 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -267,6 +267,20 @@ However the rules about "wait for reply" no longer exist; that is, all file data See examples down below, after "Discussion". +### symbolic links + +Many filesystems support symbolic links ("symlinks"). +There is no way to "send a link" in this protocol. +In the classic transfer protocol, sending a link would cause the contents of the target to be sent (using the symlink name). +As the classic transfer used "zip" to bundle directory trees, it also bundled the contents of the target (as above). + +That said, it is up to implementations to decide how to deal with symlinks. +When traversing directory trees, symlinks can point well outside of the user-selected directory and care may be required when following these. +Loops may also occur. +These problems _should_ be encountered when constructing the `DirectoryOffer`. +The receiving side will not encounter these issues as it will receive either a file or a collection of files that are all strictly below some root directory. + + ## Discussion and Open Questions {#discussion} * Overall versioning considerations From aa27570f6e18c4839bc52e3cff0825b271f6e3c2 Mon Sep 17 00:00:00 2001 From: meejah Date: Fri, 8 Jul 2022 14:24:09 -0600 Subject: [PATCH 42/49] further clarify data fragmentation etc --- dilated-file-transfer.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 900d99d..b4b3f13 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -257,9 +257,8 @@ If the receiving side responds with `OfferAccept` then (following the example ab Messages of kind `5` ("file data bytes") consist solely of file data. A single data message MUST NOT exceed 65536 bytes (65KiB) including the single byte for "kind" (so 65535 maximum payload bytes). Applications are free to choose how to fragment the file data so long as no single message is bigger than 65536 bytes. -A good default to choose in 2022 is 16KiB (2^14 - 1 payload bytes) - - XXX: what is a good default? Dilation doesn't give guidance either... +A good default to choose in 2022 is 16KiB (2^14 - 1 payload bytes). +Note the above considerations are to do with `kind=5` file-data messages only. When sending a `DirectoryOffer` each individual file is preceded by a `FileOffer` message and ends with a `FileAcknowledge` message. However the rules about "wait for reply" no longer exist; that is, all file data MUST be immediately sent (the `FileOffer`s serve as a header). @@ -267,7 +266,7 @@ However the rules about "wait for reply" no longer exist; that is, all file data See examples down below, after "Discussion". -### symbolic links +### Symbolic Links Many filesystems support symbolic links ("symlinks"). There is no way to "send a link" in this protocol. @@ -278,7 +277,7 @@ That said, it is up to implementations to decide how to deal with symlinks. When traversing directory trees, symlinks can point well outside of the user-selected directory and care may be required when following these. Loops may also occur. These problems _should_ be encountered when constructing the `DirectoryOffer`. -The receiving side will not encounter these issues as it will receive either a file or a collection of files that are all strictly below some root directory. +The receiving side will not encounter these issues as it will receive either a file or a collection of files that are all strictly below the base directory. ## Discussion and Open Questions {#discussion} @@ -330,9 +329,6 @@ Does "re-using" the `FileOffer` as a kind of "header" when streaming `DirectoryO We need _something_ to indicate the next file etc. Preliminary conclusion: it's fine and gives consistent metadata -Do the limits on message size make sense? Should "65KiB" be much smaller, potentially? -(Given that network conditions etc vary a lot, I think it makes sense for the _spec_ to be somewhat flexible here and "65k" doesn't seem very onerous for most modern devices / computers) - * compression From da4465201d8452dfd32249c6b5f22a49281019a1 Mon Sep 17 00:00:00 2001 From: meejah Date: Wed, 13 Jul 2022 16:53:02 -0600 Subject: [PATCH 43/49] verb tense --- dilated-file-transfer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index b4b3f13..6d0d929 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -62,7 +62,7 @@ In order to avoid confusion with "classic" transfer, this protocol skips "-v1" - Versions are considered as integers, so the version tag MUST always be the entire tail of the string, MUST start with `-v` and MUST end ONLY with digits (that are an integer version bigger than 0). -When multiple versions are present, a peer decides which version to use by comparing the "list of versions" that they each support and selects the highest from the intersection of these. +When multiple versions are present, a peer decides which version to use by comparing the "list of versions" that they each support and selecting the highest from the intersection of these. For example, if 3 versions existed, the two peers may present their version information like: ``` From 24e533561723e6337f2cbbc7fc71fea862e781e5 Mon Sep 17 00:00:00 2001 From: meejah Date: Wed, 13 Jul 2022 16:53:09 -0600 Subject: [PATCH 44/49] nah --- dilated-file-transfer.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 6d0d929..f180264 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -101,8 +101,6 @@ Peers MUST only send messages / attributes for features in the other side's list Since there are only the core features currently, the only valid value is an empty list. Peers MUST expect any strings in this list in the future (e.g. if a new feature is added, the protocol version isn't necessarily bumped). - XXX:: maybe just lean on "version" for now? e.g. version `2` could introduce "features"? - See "Example of Protocol Expansion" below for discussion about adding new attributes (including when we might increment the `"version"` instead of adding a new `"feature"`). From 6057b149ee47c6f3d4d32ec209efbc68212a6048 Mon Sep 17 00:00:00 2001 From: meejah Date: Wed, 13 Jul 2022 16:53:20 -0600 Subject: [PATCH 45/49] how to close the session --- dilated-file-transfer.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index f180264..31f79d7 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -119,6 +119,12 @@ They are actually an encoded `Map` with `String` keys (to use `msgpack` lingo) a All control-channel messages contain a integer "kind" field describing the sort of message it is. (That is, `"kind": "text"` for example, not the single-byte tag used for subchannel messages) +The logical "transfer session" lasts as long as the Dilation control-channel is open. +When either peer is done it closes the control-channel and the mailbox. +If one side sees the control-channel close this indicates the other peer is done; any outstanding transfers are cancelled. +Peers SHOULD prefer to cancel individual offers before closing the control channel. +The mailbox MUST only be closed after the control channel. + Rejected idea: Version message, because we already do version negotiation via mailbox features. Rejected idea: Offer/Answer messages via the control channel: we need to open a subchannel anyway and the subchannel-IDs are not intended to be part of the Dilation public API. From d1aae3871f546825aa802f896eee04a11529d543 Mon Sep 17 00:00:00 2001 From: meejah Date: Wed, 13 Jul 2022 16:53:55 -0600 Subject: [PATCH 46/49] re-word offer language to clarify modes --- dilated-file-transfer.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 31f79d7..66c60ce 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -144,9 +144,10 @@ class Message: ### Making an Offer -Either side MAY propose any number of Offers at any time after the connection is set up. -If the other peer specified `"mode": "send"` then this peer MUST NOT make any Offers. -If this peer specified `"mode": "receive"` then this peer MUST NOT make any Offers. +The ability to propose an Offer depends on the `"mode"` of each peer. +A peer with `"mode": "send"` MAY propose any number of Offers at any time after the connection is established. +A peer with `"mode": "receive"` MUST NOT propose any Offers. +A peer with `"mode": "connect"` MAY propose any number of Offers at any time unless the other peer is `"mode": "send"` in which case this peer MUST NOT propose any Offers. To make an Offer the peer opens a subchannel. Recall from the Dilation specification that subchannels are _record_ pipes (not simple byte-streams). From 74de0433210e488167e562f1d8bd5795a171c32f Mon Sep 17 00:00:00 2001 From: meejah Date: Wed, 13 Jul 2022 16:54:28 -0600 Subject: [PATCH 47/49] clarifications and minor re-wording --- dilated-file-transfer.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 66c60ce..a7a854c 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -167,7 +167,7 @@ All other byte values are reserved for future use and MUST NOT be used. XXX: maybe spec [0, 128) as reserved, and [128, 255) for "experiments"? -The sender that opened the new subchannel MUST immediately send one of the two kinds of offer messages. +The peer that opened the new subchannel MUST immediately send one of the two kinds of offer messages. To offer a single file (with message kind `1`): @@ -186,7 +186,11 @@ class FileAcknowledge: hash: bytes # 32-byte Blake2b hash of the file ``` +Although the receiving peer could determine the end-of-file itself based on the byte count it MUST wait for the `FileAcknowledge` from the sender before emitting its own `FileAcknowledge` message. + If the hash from the other peer does not match the one this peer sent, it is an error. +The user SHOULD be informed of the error. +Peers SHOULD delete the transferred bytes upon error. To offer a directory tree of many files (with message kind `2`): @@ -245,7 +249,7 @@ class OfferAccept: When the offering side gets an `OfferReject` message, the subchannel SHOULD be immediately closed. The offering side MAY show the "reason" string to the user. -When the offering side gets an `OfferAccept` message it begins streaming the file over the already-opened subchannel. +When the offering side gets an `OfferAccept` message it begins streaming the file data over the already-opened subchannel. When completed, the subchannel is closed (by the peer that made the Offer). That is, the offering side always initiates the open and close of the corresponding subchannel. @@ -273,7 +277,7 @@ See examples down below, after "Discussion". ### Symbolic Links -Many filesystems support symbolic links ("symlinks"). +Many filesystems support symbolic links (aka "symlinks"). There is no way to "send a link" in this protocol. In the classic transfer protocol, sending a link would cause the contents of the target to be sent (using the symlink name). As the classic transfer used "zip" to bundle directory trees, it also bundled the contents of the target (as above). @@ -283,6 +287,7 @@ When traversing directory trees, symlinks can point well outside of the user-sel Loops may also occur. These problems _should_ be encountered when constructing the `DirectoryOffer`. The receiving side will not encounter these issues as it will receive either a file or a collection of files that are all strictly below the base directory. +Implementations wanting to follow the classic transfer behavior should follow symlinks when descending directories and include the contents of file links (under the name of the link, not the original). ## Discussion and Open Questions {#discussion} @@ -328,7 +333,7 @@ Preliminary conclusion: centering around "the thing a human would select" (i.e. * streaming data -There is no "finished" message. Maybe there should be? (e.g. the receiving side sends back a hash of the file to confirm it received it properly?) +Added "finished" message with checksum. Does "re-using" the `FileOffer` as a kind of "header" when streaming `DirectoryOffer` contents make sense? We need _something_ to indicate the next file etc. @@ -523,6 +528,6 @@ Whenever Bob clicks "reject", his client answers with an `OfferReject` and Alice XXX what if Bob gets bored and clicks "cancel" on a file? Alice and Bob may exchange several files at different times, with either Alice or Bob being the sender. -As they wrap up the call, Bob closes his GUI client which closes the mailbox (and Dilated connection). -Alice's client sees the mailbox close. +As they wrap up the call, Bob closes his GUI client which closes the Dilation control channel and mailbox. +Alice's client sees the Dilation control-channel close. Alice's GUI tells her that Bob is done and finishes the session; she can no longer drop or add files. From 2d881ee622d991221eaafe38446ac4172ff952b7 Mon Sep 17 00:00:00 2001 From: meejah Date: Thu, 14 Jul 2022 12:41:26 -0600 Subject: [PATCH 48/49] control-message 'kind' is string --- dilated-file-transfer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index a7a854c..6fdaaa1 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -116,7 +116,7 @@ All control-channel messages are encoded using `msgpack`. Control-channel message formats are described using Python pseudo-code to illustrate the data types involved. They are actually an encoded `Map` with `String` keys (to use `msgpack` lingo) and values as per the pseudo-code. -All control-channel messages contain a integer "kind" field describing the sort of message it is. +All control-channel messages contain a "kind" field which is a string describing the sort of message it is. (That is, `"kind": "text"` for example, not the single-byte tag used for subchannel messages) The logical "transfer session" lasts as long as the Dilation control-channel is open. From 0af29f4c23d491873d1ad62ed88e93af1ec4d70a Mon Sep 17 00:00:00 2001 From: meejah Date: Thu, 14 Jul 2022 12:43:34 -0600 Subject: [PATCH 49/49] clarify the timestamp --- dilated-file-transfer.md | 1 + 1 file changed, 1 insertion(+) diff --git a/dilated-file-transfer.md b/dilated-file-transfer.md index 6fdaaa1..74ff781 100644 --- a/dilated-file-transfer.md +++ b/dilated-file-transfer.md @@ -178,6 +178,7 @@ class FileOffer: bytes: int # total number of bytes in the file ``` +The `timestamp` SHOULD be the "modification time" (that is, `st_mtime` from stat) in seconds. At the end of the file data, _both_ sides MUST send a `FileAcknowledge` message: ```python