Skip to content

Commit

Permalink
Merge pull request #1 from loomnetwork/backport-fixes-from-0.30.2
Browse files Browse the repository at this point in the history
[WIP] Backport fixes from 0.30.2
  • Loading branch information
mattkanwisher authored Apr 9, 2019
2 parents b6a6de5 + e5b4e85 commit 29bb7b1
Show file tree
Hide file tree
Showing 34 changed files with 935 additions and 168 deletions.
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ jobs:
for pkg in $(go list github.com/tendermint/tendermint/... | circleci tests split --split-by=timings); do
id=$(basename "$pkg")
GOCACHE=off go test -timeout 5m -race -coverprofile=/tmp/workspace/profiles/$id.out -covermode=atomic "$pkg" | tee "/tmp/logs/$id-$RANDOM.log"
GOCACHE=off go test -v -timeout 5m -race -coverprofile=/tmp/workspace/profiles/$id.out -covermode=atomic "$pkg" | tee "/tmp/logs/$id-$RANDOM.log"
done
- persist_to_workspace:
root: /tmp/workspace
Expand Down
74 changes: 74 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,79 @@
# Changelog

## Hacks that haven't been pushed upstream

### IMPROVEMENTS:

- [rpc] [\#2](https://github.com/mattkanwisher/tendermint/pull/2) Add `mempool_txs` RPC endpoint that provides extra info about the mempool
- [p2p] [\#1](https://github.com/mattkanwisher/tendermint/pull/1) Support custom consensus reactors
- [db] Export the constructor of the wrapped GoLevelDB iterator
- [rpc] Tweak default timeouts
- [mempool] Add metrics for tx in cache errors

### BUG FIXES:

- [mempool] [\#3](https://github.com/mattkanwisher/tendermint/pull/3) Evict txs from the mempool if they haven't been committed after `N` blocks

## Backported from v0.30.1

### BUG FIXES:

- [consensus] [\#3297](https://github.com/tendermint/tendermint/pull/3297) Flush WAL on stop to prevent data corruption during graceful shutdown.
- [consensus] [\#3310](https://github.com/tendermint/tendermint/pull/3310) Fix possible halt by resetting TriggeredTimeoutPrecommit before starting next height.
- [p2p] [\#3347](https://github.com/tendermint/tendermint/pull/3347) Secret connection check all zeroes
- [p2p] [\#3321](https://github.com/tendermint/tendermint/pull/3321) Authenticate a peer against its NetAddress.ID when dialing

## Backported from v0.29.2

### BUG FIXES:

- [node] [\#3194](https://github.com/tendermint/tendermint/pull/3194) EventBus and indexerService should be started before first block (for replay last block on handshake) execution (@ackratos)
- [p2p] [\#3247](https://github.com/tendermint/tendermint/pull/3247) Fix panic in SeedMode when calling FlushStop and OnStop concurrently
- [p2p] [\#3040](https://github.com/tendermint/tendermint/pull/3040) Fix MITM on secret connection by checking low-order points
- [mempool] [\#3221](https://github.com/tendermint/tendermint/pull/3221) Correct args order in the log msg

## Backported from v0.29.1

### IMPROVEMENTS:

- [pex] [\#3144](https://github.com/tendermint/tendermint/pull/3144) Only log "Reached max attempts to dial" once
- [rpc] [\#3197](https://github.com/tendermint/tendermint/pull/3197) Expose triggered_timeout_commit in the /dump_consensus_state


### BUG FIXES:

- [consensus] [\#3197](https://github.com/tendermint/tendermint/pull/3197) Fix consensus halt with no empty blocks from not resetting triggeredTimeoutCommit
- [p2p] [\#3150](https://github.com/tendermint/tendermint/pull/3150) Fix file descriptor leak

## Backported from v0.29.0

### BUG FIXES:

- [mempool] [\#3168](https://github.com/tendermint/tendermint/pull/3168) Limit tx size to fit in the max reactor msg size

## Can't be backported from v0.28.1 (breaks replay)

### BUG FIXES:

- [consensus] [\#3286](https://github.com/tendermint/tendermint/pull/3286) Fix consensus halt from proposing blocks with too much evidence

## Backported from v0.28.0

### IMPROVEMENTS:

- [consensus] [\#3086](https://github.com/tendermint/tendermint/issues/3086) Log peerID on ignored votes (@srmo)

### BUG FIXES:

- [p2p/conn] [\#3111](https://github.com/tendermint/tendermint/issues/3111) Make SecretConnection thread safe
- [rpc] [\#3053](https://github.com/tendermint/tendermint/issues/3053) Fix internal error in /tx_search when results are empty (@gianfelipe93)

## Backported from v0.27.4

### BUG FIXES:

- [mempool] [\#3036](https://github.com/tendermint/tendermint/issues/3036) Fix LRU cache by popping the least recently used item when the cache is full, not the most recently used one!

## v0.27.3

*December 16th, 2018*
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG_PENDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Special thanks to external contributors on this release:
### FEATURES:

### IMPROVEMENTS:
- [rpc] \#3047 Include peer's remote IP in `/net_info`

### BUG FIXES:

10 changes: 8 additions & 2 deletions consensus/common_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -124,15 +124,21 @@ func startTestRound(cs *ConsensusState, height int64, round int) {

// Create proposal block from cs1 but sign it with vs
func decideProposal(cs1 *ConsensusState, vs *validatorStub, height int64, round int) (proposal *types.Proposal, block *types.Block) {
cs1.mtx.Lock()
block, blockParts := cs1.createProposalBlock()
cs1.mtx.Unlock()
if block == nil { // on error
panic("error creating proposal block")
}

// Make proposal
polRound, propBlockID := cs1.ValidRound, types.BlockID{block.Hash(), blockParts.Header()}
cs1.mtx.RLock()
validRound := cs1.ValidRound
chainID := cs1.state.ChainID
cs1.mtx.RUnlock()
polRound, propBlockID := validRound, types.BlockID{block.Hash(), blockParts.Header()}
proposal = types.NewProposal(height, round, polRound, propBlockID)
if err := vs.SignProposal(cs1.state.ChainID, proposal); err != nil {
if err := vs.SignProposal(chainID, proposal); err != nil {
panic(err)
}
return
Expand Down
9 changes: 9 additions & 0 deletions consensus/replay.go
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,7 @@ type Handshaker struct {
stateDB dbm.DB
initialState sm.State
store sm.BlockStore
eventBus types.BlockEventPublisher
genDoc *types.GenesisDoc
logger log.Logger

Expand All @@ -209,6 +210,7 @@ func NewHandshaker(stateDB dbm.DB, state sm.State,
stateDB: stateDB,
initialState: state,
store: store,
eventBus: types.NopEventBus{},
genDoc: genDoc,
logger: log.NewNopLogger(),
nBlocks: 0,
Expand All @@ -219,6 +221,12 @@ func (h *Handshaker) SetLogger(l log.Logger) {
h.logger = l
}

// SetEventBus - sets the event bus for publishing block related events.
// If not called, it defaults to types.NopEventBus.
func (h *Handshaker) SetEventBus(eventBus types.BlockEventPublisher) {
h.eventBus = eventBus
}

func (h *Handshaker) NBlocks() int {
return h.nBlocks
}
Expand Down Expand Up @@ -432,6 +440,7 @@ func (h *Handshaker) replayBlock(state sm.State, height int64, proxyApp proxy.Ap
meta := h.store.LoadBlockMeta(height)

blockExec := sm.NewBlockExecutor(h.stateDB, h.logger, proxyApp, sm.MockMempool{}, sm.MockEvidencePool{})
blockExec.SetEventBus(h.eventBus)

var err error
state, err = blockExec.ApplyBlock(state, meta.BlockID, block)
Expand Down
11 changes: 6 additions & 5 deletions consensus/replay_file.go
Original file line number Diff line number Diff line change
Expand Up @@ -326,17 +326,18 @@ func newConsensusStateForReplay(config cfg.BaseConfig, csConfig *cfg.ConsensusCo
cmn.Exit(fmt.Sprintf("Error starting proxy app conns: %v", err))
}

eventBus := types.NewEventBus()
if err := eventBus.Start(); err != nil {
cmn.Exit(fmt.Sprintf("Failed to start event bus: %v", err))
}

handshaker := NewHandshaker(stateDB, state, blockStore, gdoc)
handshaker.SetEventBus(eventBus)
err = handshaker.Handshake(proxyApp)
if err != nil {
cmn.Exit(fmt.Sprintf("Error on handshake: %v", err))
}

eventBus := types.NewEventBus()
if err := eventBus.Start(); err != nil {
cmn.Exit(fmt.Sprintf("Failed to start event bus: %v", err))
}

mempool, evpool := sm.MockMempool{}, sm.MockEvidencePool{}
blockExec := sm.NewBlockExecutor(stateDB, log.TestingLogger(), proxyApp.Consensus(), mempool, evpool)

Expand Down
15 changes: 8 additions & 7 deletions consensus/state.go
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,6 @@ type ConsensusState struct {
// internal state
mtx sync.RWMutex
cstypes.RoundState
triggeredTimeoutPrecommit bool
state sm.State // State until height-1.

// state changes may be triggered by: msgs from peers,
Expand Down Expand Up @@ -532,6 +531,7 @@ func (cs *ConsensusState) updateToState(state sm.State) {
cs.CommitRound = -1
cs.LastCommit = lastPrecommits
cs.LastValidators = state.LastValidators
cs.TriggeredTimeoutPrecommit = false

cs.state = state

Expand Down Expand Up @@ -714,6 +714,7 @@ func (cs *ConsensusState) handleTxsAvailable() {
cs.mtx.Lock()
defer cs.mtx.Unlock()
// we only need to do this for round 0
cs.enterNewRound(cs.Height, 0)
cs.enterPropose(cs.Height, 0)
}

Expand Down Expand Up @@ -764,7 +765,7 @@ func (cs *ConsensusState) enterNewRound(height int64, round int) {
cs.ProposalBlockParts = nil
}
cs.Votes.SetRound(round + 1) // also track next round (round+1) to allow round-skipping
cs.triggeredTimeoutPrecommit = false
cs.TriggeredTimeoutPrecommit = false

cs.eventBus.PublishEventNewRound(cs.NewRoundEvent())
cs.metrics.Rounds.Set(float64(round))
Expand Down Expand Up @@ -1121,12 +1122,12 @@ func (cs *ConsensusState) enterPrecommit(height int64, round int) {
func (cs *ConsensusState) enterPrecommitWait(height int64, round int) {
logger := cs.Logger.With("height", height, "round", round)

if cs.Height != height || round < cs.Round || (cs.Round == round && cs.triggeredTimeoutPrecommit) {
if cs.Height != height || round < cs.Round || (cs.Round == round && cs.TriggeredTimeoutPrecommit) {
logger.Debug(
fmt.Sprintf(
"enterPrecommitWait(%v/%v): Invalid args. "+
"Current state is Height/Round: %v/%v/, triggeredTimeoutPrecommit:%v",
height, round, cs.Height, cs.Round, cs.triggeredTimeoutPrecommit))
"Current state is Height/Round: %v/%v/, TriggeredTimeoutPrecommit:%v",
height, round, cs.Height, cs.Round, cs.TriggeredTimeoutPrecommit))
return
}
if !cs.Votes.Precommits(round).HasTwoThirdsAny() {
Expand All @@ -1136,7 +1137,7 @@ func (cs *ConsensusState) enterPrecommitWait(height int64, round int) {

defer func() {
// Done enterPrecommitWait:
cs.triggeredTimeoutPrecommit = true
cs.TriggeredTimeoutPrecommit = true
cs.newStep()
}()

Expand Down Expand Up @@ -1526,7 +1527,7 @@ func (cs *ConsensusState) addVote(vote *types.Vote, peerID p2p.ID) (added bool,
// Not necessarily a bad peer, but not favourable behaviour.
if vote.Height != cs.Height {
err = ErrVoteHeightMismatch
cs.Logger.Info("Vote ignored and not added", "voteHeight", vote.Height, "csHeight", cs.Height, "err", err)
cs.Logger.Info("Vote ignored and not added", "voteHeight", vote.Height, "csHeight", cs.Height, "peerID", peerID)
return
}

Expand Down
124 changes: 124 additions & 0 deletions consensus/state_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1266,6 +1266,130 @@ func TestCommitFromPreviousRound(t *testing.T) {
ensureNewRound(newRoundCh, height+1, 0)
}

type fakeTxNotifier struct {
ch chan struct{}
}

func (n *fakeTxNotifier) TxsAvailable() <-chan struct{} {
return n.ch
}

func (n *fakeTxNotifier) Notify() {
n.ch <- struct{}{}
}

func TestStartNextHeightCorrectly(t *testing.T) {
config.Consensus.SkipTimeoutCommit = false
cs1, vss := randConsensusState(4)
cs1.txNotifier = &fakeTxNotifier{ch: make(chan struct{})}

vs2, vs3, vs4 := vss[1], vss[2], vss[3]
height, round := cs1.Height, cs1.Round

proposalCh := subscribe(cs1.eventBus, types.EventQueryCompleteProposal)
timeoutProposeCh := subscribe(cs1.eventBus, types.EventQueryTimeoutPropose)

newRoundCh := subscribe(cs1.eventBus, types.EventQueryNewRound)
newBlockHeader := subscribe(cs1.eventBus, types.EventQueryNewBlockHeader)
addr := cs1.privValidator.GetPubKey().Address()
voteCh := subscribeToVoter(cs1, addr)

// start round and wait for propose and prevote
startTestRound(cs1, height, round)
ensureNewRound(newRoundCh, height, round)

ensureNewProposal(proposalCh, height, round)
rs := cs1.GetRoundState()
theBlockHash := rs.ProposalBlock.Hash()
theBlockParts := rs.ProposalBlockParts.Header()

ensurePrevote(voteCh, height, round)
validatePrevote(t, cs1, round, vss[0], theBlockHash)

signAddVotes(cs1, types.PrevoteType, theBlockHash, theBlockParts, vs2, vs3, vs4)

ensurePrecommit(voteCh, height, round)
// the proposed block should now be locked and our precommit added
validatePrecommit(t, cs1, round, round, vss[0], theBlockHash, theBlockHash)

rs = cs1.GetRoundState()

// add precommits
signAddVotes(cs1, types.PrecommitType, nil, types.PartSetHeader{}, vs2)
signAddVotes(cs1, types.PrecommitType, theBlockHash, theBlockParts, vs3)
time.Sleep(5 * time.Millisecond)
signAddVotes(cs1, types.PrecommitType, theBlockHash, theBlockParts, vs4)

rs = cs1.GetRoundState()
assert.True(t, rs.TriggeredTimeoutPrecommit)

ensureNewBlockHeader(newBlockHeader, height, theBlockHash)

cs1.txNotifier.(*fakeTxNotifier).Notify()

ensureNewTimeout(timeoutProposeCh, height+1, round, cs1.config.TimeoutPropose.Nanoseconds())
rs = cs1.GetRoundState()
assert.False(t, rs.TriggeredTimeoutPrecommit, "triggeredTimeoutPrecommit should be false at the beginning of each round")
}

func TestResetTimeoutPrecommitUponNewHeight(t *testing.T) {
config.Consensus.SkipTimeoutCommit = false
cs1, vss := randConsensusState(4)

vs2, vs3, vs4 := vss[1], vss[2], vss[3]
height, round := cs1.Height, cs1.Round

partSize := types.BlockPartSizeBytes

proposalCh := subscribe(cs1.eventBus, types.EventQueryCompleteProposal)

newRoundCh := subscribe(cs1.eventBus, types.EventQueryNewRound)
newBlockHeader := subscribe(cs1.eventBus, types.EventQueryNewBlockHeader)
addr := cs1.privValidator.GetPubKey().Address()
voteCh := subscribeToVoter(cs1, addr)

// start round and wait for propose and prevote
startTestRound(cs1, height, round)
ensureNewRound(newRoundCh, height, round)

ensureNewProposal(proposalCh, height, round)
rs := cs1.GetRoundState()
theBlockHash := rs.ProposalBlock.Hash()
theBlockParts := rs.ProposalBlockParts.Header()

ensurePrevote(voteCh, height, round)
validatePrevote(t, cs1, round, vss[0], theBlockHash)

signAddVotes(cs1, types.PrevoteType, theBlockHash, theBlockParts, vs2, vs3, vs4)

ensurePrecommit(voteCh, height, round)
validatePrecommit(t, cs1, round, round, vss[0], theBlockHash, theBlockHash)

rs = cs1.GetRoundState()

// add precommits
signAddVotes(cs1, types.PrecommitType, nil, types.PartSetHeader{}, vs2)
signAddVotes(cs1, types.PrecommitType, theBlockHash, theBlockParts, vs3)
time.Sleep(5 * time.Millisecond)
signAddVotes(cs1, types.PrecommitType, theBlockHash, theBlockParts, vs4)

rs = cs1.GetRoundState()
assert.True(t, rs.TriggeredTimeoutPrecommit)

ensureNewBlockHeader(newBlockHeader, height, theBlockHash)

prop, propBlock := decideProposal(cs1, vs2, height+1, 0)
propBlockParts := propBlock.MakePartSet(partSize)

if err := cs1.SetProposalAndBlock(prop, propBlock, propBlockParts, "some peer"); err != nil {
t.Fatal(err)
}
ensureNewProposal(proposalCh, height+1, 0)

rs = cs1.GetRoundState()
assert.False(t, rs.TriggeredTimeoutPrecommit, "triggeredTimeoutPrecommit should be false at the beginning of each height")
}

//------------------------------------------------------------------------------------------
// SlashingSuite
// TODO: Slashing
Expand Down
Loading

0 comments on commit 29bb7b1

Please sign in to comment.