can we just use the hash function to flag incompatible signatures, instead of DNA/protein/etc?

[ctb]

here's a random thought for @olgabot @luizirber in particular --

right now our signatures contain entries for

"hash_function": "0.murmur64",
"molecule": "DNA",

where signatures can be flagged as incompatible due to hash function OR molecule type (or other things, like ksize). When @olgabot added dayhoff encoding, it ended up adding a whole bunch more possible incompatibilities (I'm not sure this is saved in the signature JSON currently, tho). And we're hoping to add more such things in the future, with skip-mers and other approaches.

So... I was thinking it would be possible to proliferate hash functions instead of molecule types etc.

the idea would be to create hash_functions enum types like

0.murmur64.DNA
0.murmur64.protein
0.murmur64.dayhoff

that would encode these features.

Then we could get rid of specific molecule type/other flag checks in the signatures and MinHash objects.

Thoughts?

[luizirber]

I like this idea, especially if we do an actual enum type instead of string-typing new hash functions. Are you planning to do a RFC branch, or should I?

[luizirber]

A complication: molecule and hash_function exist in different places in a signature.

[
  {
    "class": "sourmash_signature",
    "email": "",
    "hash_function": "0.murmur64",
    "license": "CC0",
    "signatures": [
      {
        "ksize": 20,
        "max_hash": 0,
        "md5sum": "98f13708210194c475687be6106a3b84",
        "mins": [],
        "molecule": "DNA",
        "num": 1,
        "seed": 42
      }
    ],
    "version": 0.4
  }
]

This complicates having DNA and protein sigs in the same file. Ideally they should be at the same level:

[
  {
    ...
    "signatures": [
      {
        "hash_function": "0.murmur64",
        "molecule": "DNA",
        ...
      }
    ],
    "version": 0.4
  }
]

... and be only one field:

[
  {
    ...
    "signatures": [
      {
        ...
        "hash_function": "0.murmur64_DNA",
      }
    ],
    "version": 0.4
  }
]

Going even further, I would like to see sourmash signatures become an object at the top-level (instead of a list), and different sketches being added to the old signatures field (which becomes sketches). The idea being that you have one signature per original dataset/file/URL, but multiple sketches to represent different aspects of the original data. Something like:

  {
    "class": "sourmash_signature",
    "license": "CC0",
    "filename": "path-to-original-file",
    "ipfs_hash": "optional, but would be cool, huh?",
    "sketches": [
      {
        ...
        "hash_function": "0.murmur64_DNA",
        "sketch_type": "minhash"
      },
      {
        ...
        "hash_function": "0.nthash_DNA",
        "sketch_type": "draff"
      },
      {
        ...
        "hash_function": "0.murmur64_DNA",
         "sketch_type": "hll"
      }
    ],
    "version": 1
  }

Relevant issues:

• Allow multiple sketch types (ksize, DNA/prot, etc.) in a database? #732
• Better conceptual definition on minhash/signature/query #616
• Potential modifications to signature format for major releases #268

Relevant PRs:

• [WIP] JSON schema #578 (I did this quickly last year for @jiarong, but we can use it to validate signatures in different versions)
• https://github.com/dib-lab/sourmash/compare/rfc/751_rust is the equivalent of [RFC] Enum for simplifying molecule checks #765 but on the Rust side this is now in Indexing in Rust #773 , but FFI still exposes is_protein/dayhoff/hp.
• [MRG] Better molecule type checks #782 (comment) a bug triggered by preprocessing not being disjoint (protein is True for aa, dayhoff and hp...)

[luizirber]

Also:

• move signature save/loading code to something closer to what SBT does, where you can still load old versions but then save into the newest one. Especially since almost all current sigs are version 0.4...
• stop with the float/string version, let's stick with integers =]

[olgabot]

Sorry I missed this earlier! I like @luizirber's suggestion of keeping the hash function in each individual signature. Would the k-mer size and scale/num be included in this as well, since those also influence the compatibility of comparing signatures?