[WIP] Draft Specification

[mikeal]

As promised, this is the new skeleton for the draft specification. It's based on the old PR with the following changes.

• Anything with an open discussion has been removed. New PR's to this branch will follow to try and resolve those issues.
• All references to CBOR have been removed. The goal of the new spec is to be codec neutral, with references to soon to be defined specifications in IPLD for generic HAMT and Binary support.
• Type definitions for simple JSON types have been added where applicable.

[kevina]

Could I have some context for these changes. Are you acting under the guidance or authority of a core member of the IPFS team.

When I wrote the first spec I was under the instructions of @whyrusleeping to use CBOR and in fact provide to provide a spec that defines the binary representation of an unixfsv2 object. Has this changed?

[mikeal]

@kevina this is based on recent conversations with @Stebalien and others.

The primary reason for using CBOR was for binary filename support. The spec will still support binary filenames it just won't require that CBOR be used to represent them. We should expect that other codecs will also have support for binary and shouldn't limit the standards we build on IPLD to only be implementable in CBOR.

In other words, the spec will require binary support but won't require CBOR.

Related: we've discussed how to standardize binary support in dag-json using something along the lines of { "/": { type: "binary", "base64": String } }.

However, if there's another reason we want to require that unixfs-v2 only be implementable in CBOR this is the place to discuss it.

[kevina]

The primary reason for using CBOR was for binary filename support.

I am not sure where you got that from. The decision from @whyrusleeping to use CBOR was well before the filename issue became apparent. The entire details of IPLD our very poorly specified in my view, which was why I struggled so much to create anything useful. Deciding that unixfsv2 will be in CBOR and then have a JSON representation from CBOR makes things concrete and easier to reason about.

{ "/": { type: "binary", "base64": String } }

Where did that come from. Is this at all related to filename support? The issue with filenames has nothing to do with JSON and all to due to the fact that in Unix filesnames are not required to to valid Unicode and hence can not be part of an IPLD path.

If there where discussion on IRC it will be extremely helpful to provide the relevant parts of conversation here.

[mikeal]

Where did that come from.

Literally just a discussion we started this week, nothing concrete yet. It's not directly related to filename support, it's just a way to standardize binary support for dag-json. It was only brought into this discussion because of the need for binary.

The issue with filenames has nothing to do with JSON and all to due to the fact that in Unix filesnames are not requested to to valid Unicode and hence can not be part of an IPLD path.

Yes, but this means that, in order to preserve the original name we need binary support in the codec.

If there where discussion on IRC it will be extremely helpful to provide the relevant parts of conversation here.

None of this happened on IRC. Discussions around unixfsv2 have been happening in threads in this repo, other IPLD repos, and in-person conversations in Berlin and elsewhere. And for months none of that turned into concrete progress.

If you'd like to use a different strategy to move things forward I'm all ears, but nothing progressed last quarter and if we want to see progress this quarter we need to try something else.

The entire details of IPLD our very poorly specified in my view, which was why I struggled so much to create anything useful.

Agreed. This is something I'm also working on. Apologies that this is happening in parallel. Because the timeline for these conversations is so long a lot of things I'd love to rely on already being in place in order to reference here are just kicking off.

Deciding that unixfsv2 will be in CBOR and then have a JSON representation from CBOR makes things concrete and easier to reason about.

This calls back to your comments about IPLD not being well specified, so let me try to unwind this a little.

The whole "JSON representation" thing has been problematic. For reference: ipld/js-ipld-dag-cbor#76 and ipld/specs#70 just to name a few.

Right now, it's assumed that IPLD codecs support at least JSON types and Links (using CIDs). Even defining a canonical representation of links in JSON is controversial.

Defining unixfs-v2 using JSON types, Links and Binary keeps us from being restricted to a single codec. What I would like to avoid is using other features of CBOR like tagging, as some have already suggested, which locks us into a codec and is a feature for which we haven't even started to discuss how to create a "JSON representation" for.

Since the one thing we've created a JSON representation for, links, isn't even on solid footing I don't see why we would want to push new work like unixfs-v2 down a path towards requiring all new representations.

There's a bigger conversation about a future type system that would further complicate all of this but in the interest of moving the spec forward I'd like to avoid relying on anything other than JSON types, Links and Binary.

I agree that we need to keep things concrete and plan to continue to update the standalone JS version of unixfs-v2, which uses CBOR, as the spec continues to evolve.