Ancient storage on archive nodes

[easeev]

Problem

Ancient storage is a great feature which works well for full mainnet nodes in particular:

/ # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/sdb                491.2G    337.6G    153.5G  69% /root
/dev/sdc                491.2G    143.9G    347.2G  29% /ancient

However, it's not remotely as good for archive mainnet node (where it will be the most valuable):

~/.ethereum/geth/chaindata # du -h
36.7G   ./ancient
3.5T    .

Both nodes are completely synced and have been running for a while.

Is there a way to improve it?

System information

Geth version: 1.9.10
OS & Version: Linux

[karalabe]

It should be of the same size on both full node and archive node. The ancient folder however only stashes away past blocks and receipts. Since the crux of the archive node database is state, that will still linger in leveldb.

[kutysam]

Hi @karalabe, our archive disk space never grows past a certain point and our logs show this crucial information every 1 minute.

ERROR[05-19|03:15:57.287] Block receipts missing, can't freeze number=5546811 hash="12262e…08051e"

I believe this is where the 'freezer' populates the ancient data folder from chaindata folder and ours seem to be stuck at block 5546811.

I understand that debug.setHead("0x593710") resets the chain to 5546800 but redownloading the remainder of the chain will take time for an archive node. Is there any other ways to repair this specific block without rewinding the headchain?

System information

Geth version: 1.9.14
OS & Version: Linux

[easeev]

Any updates on this issue @karalabe?

[karalabe]

The underlying issue was fixed, though AFAIK it should only ever happen on Clique networks (i.e. Rinkeby). I'm unsure how to fix it though without a resync unfortunately.

[easeev]

The underlying issue was fixed, though AFAIK it should only ever happen on Clique networks (i.e. Rinkeby).

We're experiencing this issue on mainnet :(