"Database compacting, degraded performance" for 2 days already

[i11m20n]

My node was already fully synced and working fine, randomly started doing that and it's not finishing or giving any progress.

System information

Geth Version: 1.10.20-stable
Git Commit: 8f2416a89a3def6ec2c749d5afafbf2c9a18e3c8
OS & Version: CentOS Linux release 8.3.2011

Parameters

nohup ./geth --syncmode=full --gcmode=archive --http --http.api=admin,debug,web3,eth,net,txpool --ws --ws.api=admin,debug,web3,eth,net,txpool --maxpeers=1000 --cache=51200 --datadir=/ethereum/.ethereum &

SSD usage

125M	./geth/nodes
7.3G	./geth/fastcache.tmp.717729320
348G	./geth/chaindata/ancient
15T	./geth/chaindata
239M	./geth/ethash
6.5G	./geth/triecache
15T	./geth
4.0K	./keystore
15T	.

log

WARN [08-02|17:11:21.802] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:11:32.075] Snapshot extension registration failed   peer=6ec80bbf err="peer connected on snap without compatible eth support"
WARN [08-02|17:12:21.823] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:13:24.801] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:13:41.630] Snapshot extension registration failed   peer=04dfa983 err="peer connected on snap without compatible eth support"
WARN [08-02|17:14:24.838] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:15:24.875] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:16:25.050] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:17:25.079] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:18:25.083] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:19:28.070] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:20:26.848] Snapshot extension registration failed   peer=f343148b err="peer connected on snap without compatible eth support"
WARN [08-02|17:20:28.185] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:21:31.097] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:22:31.113] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:23:31.125] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:24:31.139] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:25:31.152] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:25:55.117] Snapshot extension registration failed   peer=7a77d37c err="peer connected on snap without compatible eth support"
WARN [08-02|17:26:31.697] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:27:34.170] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:28:34.277] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:29:34.341] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:30:37.305] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:31:37.361] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:32:40.326] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:33:40.396] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:34:43.381] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:35:27.373] Snapshot extension registration failed   peer=3ce52055 err="peer connected on snap without compatible eth support"
WARN [08-02|17:35:43.387] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata
WARN [08-02|17:36:46.406] Database compacting, degraded performance database=/ethereum/.ethereum/geth/chaindata

What do I need to do to restore sync? Please help me, thanks.

[rjl493456442]

@i11m20n Please upgrade your node to latest version(1.10.21), in 1.10.20, we pulled some changes from go-leveldb which contains a buggy change. It will affect archive node horribly.

For more information, please check here #25413

[rjl493456442]

Please let me know if the issue is gone(eventually recovered from the compaction and can sync again).

[i11m20n]

Please let me know if the issue is gone(eventually recovered from the compaction and can sync again).

Thanks for your answer, the node is back in sync after upgrading to v1.10.21.

SSD usage has recovered from 15T to 11T.

[i11m20n]

Please let me know if the issue is gone(eventually recovered from the compaction and can sync again).

The following output often appears in the logs of v1.10.21:

INFO [08-03|17:44:15.864] Aborting state snapshot generation       root=7c2701..83c62e in=8fd7cb..881d7c at=ab702b..569e65 accounts=100,349,117 slots=414,203,755 storage=34.22GiB   dangling=0 elapsed=2h58m23.354s eta=2h19m5.606s
INFO [08-03|17:44:15.871] Resuming state snapshot generation       root=46919b..629ce1 in=8fd7cb..881d7c at=ab702b..569e65 accounts=100,349,117 slots=414,203,755 storage=34.22GiB   dangling=0 elapsed=2h58m23.361s eta=2h19m5.611s
INFO [08-03|17:44:15.889] Imported new chain segment               blocks=1   txs=22   mgas=1.926   elapsed=50.132ms     mgasps=38.409  number=15,268,774 hash=70cd91..af72fd dirty=0.00B
INFO [08-03|17:44:17.780] Deep froze chain segment                 blocks=7   elapsed=9.181ms      number=15,178,774 hash=167fec..78fc3e
INFO [08-03|17:44:23.880] Generating state snapshot                root=46919b..629ce1 at=90024e..894237 accounts=100,464,725 slots=414,315,977 storage=34.23GiB   dangling=0 elapsed=2h58m31.370s eta=2h18m49.875s
WARN [08-03|17:44:26.263] Snapshot extension registration failed   peer=7a77d37c err="peer connected on snap without compatible eth support"
INFO [08-03|17:44:31.897] Generating state snapshot                root=46919b..629ce1 in=901ca1..4edd30 at=6900d5..cb926f accounts=100,536,403 slots=414,599,291 storage=34.25GiB   dangling=0 elapsed=2h58m39.387s eta=2h18m42.513s
INFO [08-03|17:44:39.904] Generating state snapshot                root=46919b..629ce1 in=903d65..39a86b at=9e5a50..8d0345 accounts=100,626,157 slots=414,749,200 storage=34.27GiB   dangling=0 elapsed=2h58m47.394s eta=2h18m31.819s
INFO [08-03|17:44:43.241] Aborting state snapshot generation       root=46919b..629ce1 in=904b06..04f72f at=796570..257e42 accounts=100,663,115 slots=414,831,871 storage=34.28GiB   dangling=0 elapsed=2h58m50.731s eta=2h18m27.379s
INFO [08-03|17:44:43.245] Resuming state snapshot generation       root=3d5081..45ecca in=904b06..04f72f at=796570..257e42 accounts=100,663,115 slots=414,831,871 storage=34.28GiB   dangling=0 elapsed=2h58m50.735s eta=2h18m27.382s
INFO [08-03|17:44:43.387] Imported new chain segment               blocks=1   txs=311  mgas=29.993  elapsed=738.240ms    mgasps=40.628  number=15,268,775 hash=382612..e36ab0 dirty=0.00B
INFO [08-03|17:44:43.735] Chain reorg detected                     number=15,268,774 hash=70cd91..af72fd drop=1 dropfrom=382612..e36ab0 add=1 addfrom=4c055a..6fda14
INFO [08-03|17:44:43.743] Imported new chain segment               blocks=1   txs=319  mgas=29.959  elapsed=355.015ms    mgasps=84.388  number=15,268,775 hash=4c055a..6fda14 dirty=0.00B
INFO [08-03|17:44:51.252] Generating state snapshot                root=3d5081..45ecca in=906c72..23c019 at=1f5190..75f9e3 accounts=100,754,109 slots=415,002,051 storage=34.29GiB   dangling=0 elapsed=2h58m58.742s eta=2h18m16.358s
INFO [08-03|17:44:57.662] Aborting state snapshot generation       root=3d5081..45ecca in=908471..d2cc57 at=5aa81e..d83371 accounts=100,819,678 slots=415,182,050 storage=34.31GiB   dangling=0 elapsed=2h59m5.152s  eta=2h18m8.955s
INFO [08-03|17:44:57.667] Resuming state snapshot generation       root=2c5794..d4e58b in=908471..d2cc57 at=5aa81e..d83371 accounts=100,819,678 slots=415,182,050 storage=34.31GiB   dangling=0 elapsed=2h59m5.157s  eta=2h18m8.959s
INFO [08-03|17:44:57.764] Chain reorg detected                     number=15,268,774 hash=70cd91..af72fd drop=1 dropfrom=4c055a..6fda14 add=2 addfrom=22f762..8db201
INFO [08-03|17:44:57.779] Imported new chain segment               blocks=1   txs=227  mgas=15.210  elapsed=279.152ms    mgasps=54.487  number=15,268,776 hash=22f762..8db201 dirty=0.00B
INFO [08-03|17:45:05.674] Generating state snapshot                root=2c5794..d4e58b at=909e93..fdfb5f accounts=100,891,466 slots=415,393,983 storage=34.33GiB   dangling=0 elapsed=2h59m13.164s eta=2h18m1.691s
INFO [08-03|17:45:09.246] Aborting state snapshot generation       root=2c5794..d4e58b at=90aca4..4dc3fb accounts=100,929,792 slots=415,478,164 storage=34.33GiB   dangling=0 elapsed=2h59m16.735s eta=2h17m57.209s
INFO [08-03|17:45:09.248] Resuming state snapshot generation       root=86e39b..a1e679 at=90aca4..4dc3fb accounts=100,929,792 slots=415,478,164 storage=34.33GiB   dangling=0 elapsed=2h59m16.738s eta=2h17m57.211s

• Generating state snapshot
• Aborting state snapshot generation
• Resuming state snapshot generation

[rjl493456442]

@i11m20n Geth is generating the state snapshot in the background. As it said, eta=2h18m1.691s, it will still take around 2h to finish. I guess you can wait.

[i11m20n]

@i11m20n Geth is generating the state snapshot in the background. As it said, eta=2h18m1.691s, it will still take around 2h to finish. I guess you can wait.

As you said, it was completed on schedule. Thanks again for your answer!