Add async decode to bao

[ramfox]

PR: oconnor663/bao#44

[rklaehn]

We are going to do this first for ourselves, and then try to get it merged once it works.

The thing that is most urgent to implement async is the SliceDecoder .

So we are going to have some async way to do the job of a SliceDecoder in our code base, and then PR that to bao.

[dignifiedquire]

We currently don't need the SliceDecoder, just Decoder with outboard data and Encoder, so I don't quite understand why you want to start with the SliceDecoder

[rklaehn]

I think the recommended way to send around verified data is the following:

• 1. create an encoded file, either outboard or inline
• 2. use the SliceExtractor to produce a stream of a slice.
  • If you want to send the whole file, just give 0..len as range.
  • the encoding of the slice 0..len is exactly the same as the inline encoding of the entire file
• 3. then on the receiving end use the SliceDecoder to receive and incrementally verify the file

If you want to identify one of the three steps where 1. async is easiest to add and 2. the lack of async is most annoying, it is the decoder.

See the comment in oconnor663/bao#44 (comment)

If you save the "outboard" encoded file, to avoid the space overhead of an entire copy of the original input, then you can use bao slice together with the original and the outboard encoding to reproduce the "combined" encoding. That does work in streaming-friendly way, and it doesn't require seek support in the output.

[rklaehn]

I have written a completely independent slice decoder here: https://github.com/rklaehn/bao-experiment/blob/slice-decoder-2/src/lib.rs

Getting that or something with equivalent functionality based on the bao code itself added to bao will be a longer term thing.

[rklaehn]

The absolute minimum async support we are going to need is implemented in oconnor663/bao#46

There are a number of additional async things (seek support, encode support) that are nice to have but not strictly necessary. Those are in https://github.com/n0-computer/bao/ but don't need to go in for sendme 1.0

[rklaehn]

Not sure how to proceed with this. We now got our own crate bao-tree that supports all the things. Also we got abao which has an async decoder.