feat: Implement Boost equivalent deal market in Curio

[LexLuthr]

How the new market works:

Modules:

Libp2p

Takes in all minerIDs and start a libp2p node for each. Table for storing address, key etc

MK12 Deal handler

Starts listening to deal protocols on each libp2p and then forwards the incoming requests to MK12 market module.

Deal market

Main poller process which starts all relevant modules based on config. For now, we only have one module Mk12. This module exposes a method called ExecuteDeal(). This method is used by Libp2p handler to forward the deal or execution from libp2p.
Deal market also load the ingestor modules (snap and normal deal flow) which assigns the deals from market to sectors

Market tasks

We have 3 different tasks in market with their own pollers based on a deal pipeline state

1. CommP - Performs commP using remote read of pieces for all deals
2. PSD - send a PSD message for a set of deals per minerID
3. FindDeal - Finds and assigns the on chain deal ID to each deals after PSD

Indexing

Indexing task is a IAmBored. The entry is made in a dedicated table after MoveStorage task and this in turn triggers indexing of the deal. Once indexed, deal can be removed from the mk12 deal pipeline and piece/deal metadata is updated.

Index Store

A new store which exclusively is used for indexing based on Cassandra.

Pending

• When to start offline deal

  • Should there be a manual trigger for all offline deals
  • Should we have import-data equivalent

• Index-Provider

  • We should probably do our own version instead of using index-provider library
  • lib-ipni should have all we need in communication protocols

• Libp2p options

Deal Flow

Deal on Libp2p > libp2p hanlder (handle the deal) > Call MK12 modules inside the Deal market > Work through Market tasks > Deal market adds to sector pipeline using Ingestor > Create Indexing task (Move Storage) > Index > GC cleanup of deal pipeline

TODO:

• Add description of functionality in PR
• Add UI
• Figure out Offline deal trigger (? import from file ?)
• Deal retry with new sector
• Clean up
• comment config
• Add IndexStore tests
• Add docs
• Remove GetIndex
• Use Curio cfg in all new funcs
• Should we clean up market_offline_urls table after a URL is used? Should this be generic or deal specific?
• Complete libp2p setup
• Clean up failed deal/fix GC
• MarketAddBalance for SP side with PSD wallet
• Backpressure

Note

Remaining feature will be added in separate PRs. This is ready for review and merge.

[magik6k]

First pass, in general this is moving in the right direction.

I'm really excited to start testing this when we get closer to completion, it will make my life so, so much easier.

[magik6k]

We might want to add some flag to piece refs / piece park pieces which says that the piece is coming from an external, potentially untrusted source, eventually adding more special handling here.

(Also eventually one day we should add aria2 support here)

[magik6k]

Also kinda tangential, but would be really cool to have a column in the parked_pieces table for download progress updated e.g. every 5s

[magik6k]

Wow this is big

Generally I'd say it's looking really good. I expect that as we run this some things will need adjusting, but on a high level this looks quite solid.

Lots of comments are more on the nicpicky side, but there are some that would be nice to address.

Overall feel free to merge into the base branch, the sooner we get this running in the real world the sooner we'll get this shipped.

[magik6k]

Honestly at this point I wouldn't do further reviews, looks really good.

Some minor comments, but other than that feel free to merge into the staging branch