sourmash gather is very slow on large, diverse metagenomes

[ctb]

Not entirely sure what's going on, but some thoughts --

• top levels of SBT are probably over-full, should look at larger SBT tables;
• maybe support diagnostic output (number of tables searched, etc.) in some output file;
• we should mention the use of gather --scaled directly in the docs somewhere, if we don't already;
• support progressive output (have gather output stuff progressively to disk #258) so cancelling/killing isn't so painful
• support adaptive downsampling with gather (speed up sbt_gather by trial downsampling? #222)
• maybe provide a smaller/lower-scaled SBT (--scaled 100000) for people to use

cc @ljcohen

[mytluo]

Hi @ctb,

Would it be possible to parallelize sourmash gather? That is, if there is an option to break up the output of compute into multiple .sig files, then run each .sig file in parallel for sourmash gather. Not sure if this is feasible in any way, we were curious if it would be.

[ctb]

hi @mytluo alas, it's not particularly amenable to parallelization. We have a few more ideas up our sleeves to make it faster, not sure if any of them will work.

[ctb]

See #519 for a list of our failures, along with one success :)

[ctb]

See #615 for some significant performance improvements; more coming.