Implement more/better revindex functionality on top of LCA databases.

[ctb]

With the merge of #533, the LCA databases (should) now contain the full set of hashes that SBTs do: previously, LCA DBs were buggy and contained a somewhat random collection of hashes that were connected with taxonomic IDs, but now they have everything whether or not it has a tax id. There's a lot of opportunity to connect back to the revindex code that @halexand and I have been using for various things, viz https://github.com/ctb/2017-sourmash-revindex, and make that code better/more complete/more usable/integrated into sourmash, either as an extension or directly in sourmash.

Just a thought :)

[ctb]

more thoughts, based on prodding from @bluegenes to think about renaming "LCA databases" :)

I think the right medium term thing to do is:

• split LCA databases into "revindex" and "taxinfo";
• rename LCA database format to revindex;
• (somehow) enable the provision of taxinfo to sourmash search and sourmash gather routines, so that we can add taxinfo-based output and analyses on top of any operation;

but I don't want to delay 2.0 for this; I think it's a 3.0-ish kind of thing.

[ctb]

Some more thoughts!

If we make taxinfo databases include a taxonomic hierarchy as well (see taxlist in sourmash/lca/lca_utils.py) then we can make this work with LINS as well as AQU's taxonomy and NCBI's taxonomy.

[ctb]

Digging into the revindex code, it looks like the three main pieces of functionality we'd want in LCA_Database are:

• tracking abundances, per https://github.com/ctb/2017-sourmash-revindex/blob/master/extract-hashvals-by-sample.py
• filtering out rare hashes (ref add functionality to sourmash signature to extract common hashes? #603) - taylor uses https://github.com/taylorreiter/cosmo-kmers/blob/master/scripts/hashes-to-numpy-2.py
• adding in some form of clustering-by-hashes support, cf https://github.com/ctb/2017-sourmash-revindex/blob/master/hashes-to-association.py or https://github.com/ctb/2017-sourmash-revindex/blob/master/hashes-to-numpy-2.py. The right code for this may be in charcoal, charcoal/utils.py::make_distance_matrix.

we might also be interested in incorporating code from https://github.com/ctb/2017-sourmash-revindex/blob/master/classify-common-hashes.py to do some kind of full-database classification.

[luizirber]

tracking abundances, per https://github.com/ctb/2017-sourmash-revindex/blob/master/extract-hashvals-by-sample.py

An initial way of doing this would be storing (idx, abund) for each hashval. Not terribly efficient, but...

(long term research project: go super fancy and implement a REINDEER index, but not today =])

[ctb]

#1013 adds protein, dayhoff, and hp signature indexing in LCA databases.

[ctb]

#1015 adds abundance tracking into LCA databases. (CLOSED / NOT MERGED)

[ctb]

it would be easy to support storing abundances in SqliteIndex ref #1808, but we may decide not to do it just yet.

[ctb]

this idea lives on in mastiff, and also we've moved away from recommending LCA based approaches for taxonomy. So I'm closing.

ref #2760 for arguments in favor of using gather+tax rather than LCA.