Synching speed is too slow

[Fraccaman]

Summary of Bug

Im trying to sync a node but its taking three+ weeks. Sync speed is now 1block/s. Height is ~6106327. The configuration files Im using can be found here: https://github.com/heliaxdev/ibc-setup/tree/master/stargate. The machine shouldn't the bottleneck.

Version

gaiad version -> v4.2.1

Steps to Reproduce

You can follow the README.md here: https://github.com/heliaxdev/ibc-setup

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[tac0turtle]

What sort of machine are using, how many peers do you have, do all your peers have the nessecary blocks?
1 block/sec is extremely slow, it can be induced by multiple things

[Fraccaman]

Im using an C5.xlarge from aws with 500gb io1 disk (its a 4core, 8gb ram with iops optimized disk). I have set this option in config.toml to have some seed nodes. I guess the gaia is taking care of finding other peers, right?

[shahankhatch]

This is the case for the current network state. The only way to reach tip is to download a recent snapshot from https://cosmos.quicksync.io/.

[shahankhatch]

afaict this sync rate has no relationship to peers, hardware, or network bandwidth; it's just the current block execution mechanism

[tac0turtle]

afaict this sync rate has no relationship to peers, hardware, or network bandwidth; it's just the current block execution mechanism

If your peers don't have the blocks, then you won't be able to sync fast. Fast sync downloads blocks from multiple peers in parallel, if only one peer has block 1-500 then you are only downloading one block at a time, hence slow syncing times.

Unfortunately, Tendermint does not have service discovery, so for the time being we have to put up with this.

[Fraccaman]

So, either I download a snapshot or this is the current sync speed?

[tac0turtle]

How many peers do you have?

[Fraccaman]

Sorry, I dont know how to check that :( Can I check it thought some rpc endpoint?

[Fraccaman]

Also I have another question.
Sending a request to the /status node endpoint returns the following response:

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {
    "node_info": {
      "protocol_version": {
        "p2p": "8",
        "block": "11",
        "app": "0"
      },
      "id": "48f9939891cc0bcc251f8c9a076170ca2a1bbf4b",
      "listen_addr": "tcp://0.0.0.0:26656",
      "network": "cosmoshub-4",
      "version": "v0.34.9",
      "channels": "40202122233038606100",
      "moniker": "heliax-1",
      "other": {
        "tx_index": "on",
        "rpc_address": "tcp://127.0.0.1:26657"
      }
    },
    "sync_info": {
      "latest_block_hash": "619993B149C349D9D06682530BB9C22C542B1623C4303EACAD068C1EB4ADBED4",
      "latest_app_hash": "5B5752A7221FFA4C83E874B08F290BC8612C1CD3FDFDBB3358EFA07524687920",
      "latest_block_height": "6180421",
      "latest_block_time": "2021-05-11T15:54:33.191696391Z",
      "earliest_block_hash": "1455A0C15AC49BB506992EC85A3CD4D32367E53A087689815E01A524231C3ADF",
      "earliest_app_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855",
      "earliest_block_height": "5200791",
      "earliest_block_time": "2019-12-11T16:11:34Z",
      "catching_up": false
    },
    "validator_info": {
      "address": "4A78F46B00A73D7CE98D3B8501A1A451E66921AD",
      "pub_key": {
        "type": "tendermint/PubKeyEd25519",
        "value": "VQbyK9h63RDkrFEM1WGudkDyUYCoIBOYTOsNIxUfaBY="
      },
      "voting_power": "0"
    }
  }
}

According to https://cosmos.bigdipper.live the last block is 6,535,273. So, why is catching_up -> false in the response ?

[faddat]

OK so here's my machine spec:

CPU: 32 core threadripper
RAM: 128GB
OS: Arch Linux
Disks: 6x 2tb pcie 4.0 nvme in raid 0

I can confirm this report. I am not quite at 1 block per second, but it is very slow. Gaia is notably slower than other Cosmos-SDK chains.

I may try to state sync via the public RPC's but I hear that is hit and miss too.