Skip to content

Commit

Permalink
rewrite to use tx hash dictionary
Browse files Browse the repository at this point in the history
  • Loading branch information
@yihuang
yihuang committed Dec 4, 2023
1 parent 2c32a75 commit a2df33e
Showing 1 changed file with 53 additions and 129 deletions.
182 changes: 53 additions & 129 deletions docs/architecture/adr-069-unordered-account.md
View file
Original file line number Diff line number Diff line change
@@ -1,194 +1,118 @@
# ADR 069: Un-Ordered Nonce Lane
# ADR 070: Un-Ordered Transaction Inclusion

## Changelog

* Nov 24, 2023: Initial Draft
* Dec 4, 2023: Initial Draft

## Status

Proposed

## Abstract

We propose to add an extra nonce "lane" to support un-ordered (concurrent) transaction inclusion.
We propose a way to do replay-attack protection without enforcing the order of transactions, it don't use nonce at all. In this way we can support un-ordered transaction inclusion.

## Context

As of today, the nonce value (account sequence number) prevents replay-attack and ensures the transactions from the same sender are included into blocks and executed in sequential order. However it makes it tricky to send many transactions concurrently in a reliable way. IBC relayer and crypto exchanges would be typical examples of such use cases.
As of today, the nonce value (account sequence number) prevents replay-attack and ensures the transactions from the same sender are included into blocks and executed in sequential order. However it makes it tricky to send many transactions from the same sender concurrently in a reliable way. IBC relayer and crypto exchanges are typical examples of such use cases.

## Decision

Add an extra un-ordered nonce lane to support un-ordered transaction inclusion, it works at the same time as the default ordered lane, the transaction builder can choose either lane to use.
We propose to add a boolean field `unordered` to transaction body to mark "un-ordered" transactions.

The un-ordered nonce lane accepts nonce values that is greater than the current sequence number plus 1, which effectively creates "gaps" of nonce values, those gap values are tracked in the account state, and can be filled by future transactions, thus allows transactions to be executed in a un-ordered way.
Un-ordered transactions will bypass the nonce rules and follow the rules described below instead, in contrary, the default transactions are not impacted by this proposal, they'll follow the nonce rules the same as before.

It also tracks the last block time when the latest nonce is updated, and expire the gaps certain timeout reached, to mitigate the risk that a middleman intercept an old transaction and re-execute it in a longer future, which might cause unexpected result.
When an un-ordered transaction are included into block, the transaction hash is recorded in a dictionary, new transactions are checked against this dictionary for duplicates, and to prevent the dictionary grow indefinitly, the transaction must specify `timeout_height` for expiration, so it's safe to removed it from the dictionary after it's expired.

The design introducs almost zero overhead for users who don't use the new feature.
The dictionary can be simply implemented as an in-memory golang map, a preliminary analysis shows that the memory consumption won't be too big, for example `32M = 32 * 1024 * 1024` can support 1024 blocks where each block contains 1024 unordered transactions. For safty, we should limit the range of `timeout_height` to prevent very long expiration, and limit the size of the dictionary.

### Transaction Format

Add a boolean field `unordered` to the `BaseAccount` message, when it's set to `true`, the transaction will use the un-ordered lane, otherwise the default ordered lane.

```protobuf
message BaseAccount {
message TxBody {
...
uint64 account_number = 3;
uint64 sequence = 4;
boolean unordered = 5;
}
```

### Account State

Add some optional fields to the account state:

```golang
type UnorderedNonceManager struct {
Sequence uint64
Timestamp uint64 // the block time when the Sequence is updated
Gaps IntSet
}
type Account struct {
// default to `nil`, only initialized when the new feature is first used.
unorderedNonceManager *UnorderedNonceManager
boolean unordered = 4;
}
```

The un-ordered nonce state includes a normal sequence value plus the set of unused(gap) values in recent history, these recorded gap values can be reused by future transactions, after used they are removed from the set and can't be used again, the gap set has a maximum capacity to limit the resource usage, when the capacity is reached, the oldest gap value is removed, which also makes the pending transaction using that value as nonce will not be accepted anymore.

### Nonce Validation Logic
### Ante Handlers

The prototype implementation use a roaring bitmap to record these gap values, where the set bits represents the the gaps.
Bypass the nonce decorator for un-ordered transactions.

```golang
// CheckNonce switches to un-ordered lane if the MSB of the nonce is set.
func (acct *Account) CheckNonce(nonce uint64, unordered bool, blockTime uint64) error {
if unordered {
if acct.unorderedNonceManager == nil {
acct.unorderedNonceManager = NewUnorderedNonceManager()
}
return acct.unorderedNonceManager.CheckNonce(nonce, blockTime)
func (isd IncrementSequenceDecorator) AnteHandle(ctx sdk.Context, tx sdk.Tx, simulate bool, next sdk.AnteHandler) (sdk.Context, error) {
if tx.UnOrdered() {
return next(ctx, tx, simulate)
}

// current ordered nonce logic
// the previous logic
}
```

### UnorderedNonceManager
A decorator for the new logic.

```golang
type TxHash [32]byte

const (
// GapsCapacity is the capacity of the set of gap values.
GapsCapacity = 1024
// MaxGap is the maximum gaps a new nonce value can introduce
MaxGap = 1024
// GapsExpirationDuration is the duration in seconds for the gaps to expire
GapsExpirationDuration = 60 * 60 * 24
)
// MaxNumberOfTxHash * 32 = 128M max memory usage
MaxNumberOfTxHash = 1024 * 1024 * 4

// getGaps returns the gap set, or create a new one if it's expired
func(unm *UnorderedNonceManager) getGaps(blockTime uint64) *IntSet {
if blockTime > unm.Timestamp + GapsExpirationDuration {
return NewIntSet(GapsCapacity)
}
// MaxUnOrderedTTL defines the maximum ttl an un-order tx can set
MaxUnOrderedTTL = 1024
)

return &unm.Gaps
type DedupTxDecorator struct {
hashes map[TxHash]struct{}
}

// CheckNonce checks if the nonce in tx is valid, if yes, also update the internal state.
func(unm *UnorderedNonceManager) CheckNonce(nonce uint64, blockTime uint64) error {
switch {
case nonce == unm.Sequence:
// special case, the current sequence number must have been occupied
return errors.New("nonce is occupied")

case nonce > unm.Sequence:
// the number of gaps introduced by this nonce value, could be zero if it happens to be `unm.Sequence + 1`
gaps := nonce - unm.Sequence - 1
if gaps > MaxGap {
return errors.New("max gap is exceeded")
}

gapSet := unm.getGaps(blockTime)
// record the gaps into the bitmap
gapSet.AddRange(unm.Sequence + 1, unm.Sequence + gaps + 1)

// update the latest nonce
unm.Gaps = *gapSet
unm.Sequence = nonce
unm.Timestamp = blockTime

default:
// `nonce < unm.Sequence`, the tx try to use a historical nonce
gapSet := acct.getGaps(blockTime)
if !gapSet.Contains(nonce) {
return errors.New("nonce is occupied or expired")
}

gapSet.Remove(nonce)
unm.Gaps = *gapSet
func (dtd *DedupTxDecorator) AnteHandle(ctx sdk.Context, tx sdk.Tx, simulate bool, next sdk.AnteHandler) (sdk.Context, error) {
// only apply to un-ordered transactions
if !tx.UnOrdered() {
return next(ctx, tx, simulate)
}
return nil
}

// IntSet is a set of integers with a capacity, when capacity reached, drop the smallest value
type IntSet struct {
capacity int
bitmap roaringbitmap.BitMap
}

func NewIntSet(capacity int) *IntSet {
return &IntSet{
capacity: capacity,
bitmap: *roaringbitmap.New(),
if tx.TimeoutHeight() == 0 {
return nil, errorsmod.Wrap(sdkerrors.ErrLogic, "unordered tx must set timeout-height")
}
}

func (is *IntSet) Add(n uint64) {
if is.bitmap.GetCardinality() >= is.capacity {
// drop the smallest one
is.bitmap.Remove(is.bitmap.Minimal())
if tx.TimeoutHeight() > ctx.BlockHeight() + MaxUnOrderedTTL {
return nil, errorsmod.Wrapf(sdkerrors.ErrLogic, "unordered tx ttl exceeds %d", MaxUnOrderedTTL)
}

is.bitmap.Add(n)
}

// AddRange adds the integers in [rangeStart, rangeEnd) to the bitmap.
func (is *IntSet) AddRange(start, end uint64) {
n := end - start
if is.bitmap.GetCardinality() + n > is.capacity {
// drop the smallest ones until the capacity is not exceeded
toDrop := is.bitmap.GetCardinality() + n - is.capacity
for i := uint64(0); i < toDrop; i++ {
is.bitmap.Remove(is.bitmap.Minimal())
if !ctx.IsCheckTx() {
// a new tx included in the block, add the hash to the dictionary
if len(dtd.hashes) >= MaxNumberOfTxHash {
return nil, errorsmod.Wrap(sdkerrors.ErrLogic, "dedup map is full")
}
dtd.hashes[tx.Hash()] = struct{}
} else {
// check for duplicates
if _, ok := dtd.hashes[tx.Hash()]; ok {
return nil, errorsmod.Wrap(sdkerrors.ErrLogic, "tx is dupliated")
}
}

is.bitmap.AddRange(start, end)
return next(ctx, tx, simulate)
}
```

func (is *IntSet) Remove(n uint64) {
is.bitmap.Remove(n)
}
### Start Up

func (is *IntSet) Contains(n uint64) bool {
return is.bitmap.Contains(n)
}
```
On start up, the node needs to re-fill the tx hash dictionary by scanning `MaxUnOrderedTTL` number of historical blocks for un-ordered transactions.

An alternative design is to store the tx hash dictionary in kv store, then no need to warm up on start up.

## Consequences

### Positive

* Support concurrent transaction inclusion.
* Only optional fields are added to account state, no state migration is needed.
* No runtime overhead when the new feature is not used.
* Support un-ordered and concurrent transaction inclusion.

### Negative

- Some runtime overhead when the new feature is used.
- Start up overhead to scan historical blocks.

## References

Expand Down

0 comments on commit a2df33e

Please sign in to comment.