Research: v5 transaction format

[bitjson]

Just opening this issue to have a URL to share until I and/or another contributor get around to implementing a demo of cross-input signature aggregation in detached signatures, presumably as part of a v5 transaction format proposal. Also implicit/optional sequence numbers (but not locktime to avoid disincentivizing re-org resistance), probably some solution for fractional satoshis, maybe implied OP_RETURNs, and compactUint everything else. (And consider enabling or requiring v5 TXs to support UTXO Hash Sets.)

A key goal is to reduce the size/cost of CashFusion transactions, e.g. the largest one so far was ~40KB, with 261 inputs and 114 outputs¹. Replacing each signature in this TX with a reference to a detached signature (e.g. OP_1) would save 261 * 64 = 16,704 bytes (minus the aggregated, detached signature itself), bringing the transaction size down to ~24KB (41% savings). The savings are also greater for transactions with more inputs than outputs, so the overall effect will also encourage (privacy-preserving) UTXO set consolidation.

Also worth noting that we could save an additional 261 * 33 = 8,613 bytes (down to ~15KB, for 62% total savings) if all inputs switched to P2PK rather than P2PKH. Between the funding output (the change output of the last TX) and fusion input, each user saves 24 bytes of public key hash and P2PKH overhead.

There are some privacy considerations on the P2PK usage: you don't want to leak which output is the "change" if you're paying to P2PKH, but if the funding transaction was already using PayJoin, there was no meaningful cost to negotiating P2PK addresses rather than P2PKH. We can also expect 1) network wide P2PK usage to increase due to the cost savings and 2) if some popular wallets support both P2PK and P2PKH (esp. mixed use) the difference stops being useful as a privacy-breaking heuristic (wallets could even juggle funds between them to throw off naive trackers).

Chaingraph Queries

Paste into https://try.chaingraph.cash/:

 {
  transaction(
    where: {
      hash: {
        _eq: "\\xf55237e16408134f6ff21c75c857f0db38dc106a7bf2572af3717475cbefdf02"
      }
    }
  ) {
    hash
    input_count
    output_count
    size_bytes
    encoded_hex
}

Result:

{
  "data": {
    "transaction": [
      {
        "hash": "\\xf55237e16408134f6ff21c75c857f0db38dc106a7bf2572af3717475cbefdf02",
        "input_count": "261",
        "output_count": "114",
        "size_bytes": "40703",
        "encoded_hex": "[... snip ... ]"
      }
    ]
  }
}

Note: until we improve performance of this sort of query, you need a trusted instance to find/filter CashFusion transactions:

query CashFusionTxs {
  search_output_prefix(
    args: { locking_bytecode_prefix_hex: "6a0446555a00" }
    limit: 100
  ) {
    transaction_hash
    transaction {
      input_count
      output_count
      size_bytes
      block_inclusions {
        block {
          hash
          height
        }
      }
    }
  }
}

[bitjson]

Latest spitball for v5 TXs (given Limits/BigInt work):

• Both fractional satoshis and fractional cashtokens could be represented by two CompactUints - first is whole units, second is a numerator with the CompactUint length indicating the denominator, e.g. 0x40 is 64/256 (0.25), 0x80 is 0.5, 0xc0 is 0.75, up to 0xfc (252/256, 0.984375), then up to 0xfdffff (65535/65536, ~= 0.999985), 0xfeffffffff (4294967295/4294967296 ~= 0.99999999977), and 9*ff (18446744073709551615n/18446744073709551616n, that's 0.999... to 19 decimal places).
  • This optimally-minimizes byte length for representing evenly-divided precisions (i.e. splitting values between outputs), and allows another satoshis_or_tokens * 18446744073709551616n of precision with minimal complexity in balancing TX inputs/outputs using only 64-bit math.
  • Note that this format makes representing some base-10 fractions a little messier (0.1 is the classic example): good approximations for .10 are 0x1a (26/256=0.1015625), 0xfd9a19 (6554/65536 = 0.1000061035), 0xfe99999919 (429496730/4294967296 ~= 0.10000000009313226), or even ff9a99999999999919 (18446744073709551620000000000000000000n / 18446744073709551616n = 1000000000000000000n, that's .1000... to 19 decimal places) but it's easy to select/format for human consumption, e.g. in JS: (26/256).toFixed(2) = '0.10'. Even for these non-base-2 amounts, the overall cost of representing a satoshi amount is typically better than today's highly inefficient format. E.g. 1.1 satoshis is 0x011a (1.1015625 sats) or even 0x01FE99999919 (1.1 to 10 decimal places) as compared to today's whole-unit single satoshi encoding of: 0x0100000000000000. (And remember, at these precisions, humans are not typing base-10 numbers: human-typed numbers are being converted, often through an exchange rate and a UTXO selection algorithm, to the best choice by wallet software, optimizing for fees by minimizing transaction byte-length.)
• New OP_UTXOTOKENAMOUNTPRECISE, OP_OUTPUTTOKENAMOUNTPRECISE, OP_UTXOVALUEPRECISE, and OP_OUTPUTVALUEPRECISE opcodes could return values as BigInts of maximum precision (existing operations should continue returning whole units; critical for contracts wanting to operate within more strictly enforced boundaries, e.g. some market making algorithms).
• New fungible CashToken mints can either continue to exclude the extra precision, or with another TOKEN_PREFIX bit set, include the "fractional" CompactUint on all token amounts. (Existing token systems can create trustless migration covenants, but we can't add precision to existing cashtokens without breaking the possible expectation of indivisibility; also, a lot of fungible token systems probably don't want or need the extra divisibility, and saving 1-byte per output would continue to be a feature of "indivisible" cashtokens.)
• Detached signatures after locktime (no need to bundle with a protocol-level signature aggregation algorithm, they're useful already, and a future upgrade can lock down a non-interactive aggregation system) or 0x00 if none.
• 0 amounts imply OP_RETURN
• Compress everything else (except locktime) with CompactUints.

[bitjson]

Adding to spitball (revisiting byte saving ideas from here):

• Allow non-push opcodes in unlocking bytecode: this would create a malleability concern in v1 and v2 transactions, but since v5 transactions can optionally eliminate malleability with detached signatures, it's actually reasonable to offer users both features at the same time.
  • This would allow e.g. giant P2PKH address sweep transactions to specify <0> <key.public_key> in the zeroth input (where the 0 references the first detached signature), and all later inputs would have a 6-byte unlocking bytecode: <0> <0> OP_INPUTBYTECODE <2> OP_SPLIT OP_NIP (0x00ca527f77). Saving 94 bytes per input after the first input: 65-byte schnorr signature, plus 33-byte compressed public key, plus 2 bytes of push opcodes, minus the 6-byte pattern.
  • Savings for more complex contracts also rack up too – similar P2SH20 and P2SH32 UTXOs can be spent by pushing most or all of the redeem bytecode only once, with later inputs referencing it. (And note that the availability of this feature would allow contract authors to design with it in mind, optimizing final TX sizes.)
  • OP_EVAL would generally save some bytes of parsing operations (e.g. 4 bytes above), esp. for inputs that truly need only to copy the exact contents of another input (copying up to 16 inputs is 3 bytes, up to 127 is 4 bytes).
  • We could consider some sort of "detached data" annex that allows for arbitrary data to be referenced (rather than just signatures), but that adds some meaningful additional complexity around control of the annex contents (locking bytecode can't really use it safely, since it needs to commit to things in advance – any use of such an annex would be very similar to just using the stack) and in practice, referencing into such an annex is going to cost the same as referencing into various other inputs. So really, wallets can just deposit whatever data they need to reference into inputs according to some optimization strategy (imagine wanting to deduplicate references to several public keys across a variety of inputs: the wallet can just stick each public key into a different input and inspect/parse it from there later across all other inputs). It's hard (and certainly messier at the protocol level) to save more bytes with such an annex vs. simply adding OP_EVAL.
  • And I'll also note that these savings are useful beyond just being "compression with extra steps" – even if e.g. the P2P protocol gzipped everything, we don't operate on compressed objects at the user layer: all hashes are digests of uncompressed transactions and block headers, blockchain indexers and analysis APIs need to index uncompressed results for actual use, VMs operate on uncompressed bytecode (and naive decompression could be a DoS vector), etc. In practice, savings at this layer are a lot more useful than just reducing the bytes that go over the wire.