From df7ffc2ef203e55514e439597b7f6136002414e6 Mon Sep 17 00:00:00 2001 From: Ian Harris Date: Thu, 28 Apr 2022 10:58:16 +0100 Subject: [PATCH 1/4] docs: add snapshot instructions --- docs/docs/joining-a-testnet.md | 2 +- docs/docs/networks.md | 33 +++++++++++---------- docs/docs/snapshots.md | 52 ++++++++++++++++++++++++++++++++++ 3 files changed, 71 insertions(+), 16 deletions(-) create mode 100644 docs/docs/snapshots.md diff --git a/docs/docs/joining-a-testnet.md b/docs/docs/joining-a-testnet.md index bbbf101e..1c3cb2f3 100644 --- a/docs/docs/joining-a-testnet.md +++ b/docs/docs/joining-a-testnet.md @@ -67,7 +67,7 @@ curl https://rpc-capricorn.fetch.ai:443 | jq '.result.genesis' > ~/.fetchd/confi fetchd start --p2p.seeds=fec822ecf6e503a694a709ce663fd0c6da5fda3e@connect-capricorn.fetch.ai:36956 ``` -Your local node will then start to synchronise itself with the network, replaying all blocks and transactions up to the current block. Depending on the age of the network and your hard disk speed, this could take a while. +Your local node will then start to synchronise itself with the network, replaying all blocks and transactions up to the current block. Depending on the age of the network and your hard disk speed, this could take a while. Consider using [chain snapshots](../snapshots/) to speed up this process. To know when your node as finished syncing, you can query it's status from its RPC API: diff --git a/docs/docs/networks.md b/docs/docs/networks.md index d87b8e13..0843ebea 100644 --- a/docs/docs/networks.md +++ b/docs/docs/networks.md @@ -17,8 +17,9 @@ The chain identifier of our production network is fetchhub-4. | REST Endpoint | | | Block Explorer | [https://explore-fetchhub.fetch.ai](https://explore-fetchhub.fetch.ai) | | Token Faucet | N/A | -| Genesis | `curl https://raw.githubusercontent.com/fetchai/genesis-fetchhub/fetchhub-4/fetchhub-4/data/genesis_migrated_5300200.json --output ~/.fetchd/config/genesis.json` | +| Genesis | `curl https://raw.githubusercontent.com/fetchai/genesis-fetchhub/fetchhub-4/fetchhub-4/data/genesis_migrated_5300200.json --output ~/.fetchd/config/genesis.json` | | Seed Node(s) | 17693da418c15c95d629994a320e2c4f51a8069b@connect-fetchhub.fetch.ai:36456,a575c681c2861fe945f77cb3aba0357da294f1f2@connect-fetchhub.fetch.ai:36457,d7cda986c9f59ab9e05058a803c3d0300d15d8da@connect-fetchhub.fetch.ai:36458 | +| Snapshots |

| ## Mainnet Archives @@ -99,17 +100,19 @@ This network is running the same software as our mainnet (`fetchhub-3`), and is This network is used for testing the future upgrade to mainnet. -| Parameter | Value | -| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Chain ID | dorado-1 | -| Denomination | atestfet | -| Decimals | 18 (1testfet = 1000000000000000000atestfet) | -| Min Gas Prices | 1000000000atestfet | -| Version | [v0.10.x (fetchd >= v0.10.x)](https://github.com/fetchai/fetchd/releases/tag/v0.10.0) | -| RPC Endpoint | | -| GRPC Endpoint | | -| REST Endpoint | | -| Block Explorer | [https://explore-dorado.fetch.ai/](https://explore-dorado.fetch.ai/) | -| Ledger Explorer | [https://browse-dorado.fetch.ai/](https://browse-dorado.fetch.ai/) | -| Token Faucet | Use block explorer | -| Seed Node(s) | b9b9717975b49a57e62ea93aa4480e091ae0660@connect-dorado.fetch.ai:36556,46d2f86a255ece3daf244e2ca11d5be0f16cb633@connect-dorado.fetch.ai:36557,066fc564979b1f3173615f101b62448ac7e00eb1@connect-dorado.fetch.ai:36558 | +| Parameter | Value | +| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Chain ID | dorado-1 | +| Denomination | atestfet | +| Decimals | 18 (1testfet = 1000000000000000000atestfet) | +| Min Gas Prices | 1000000000atestfet | +| Version | [v0.10.x (fetchd >= v0.10.3)](https://github.com/fetchai/fetchd/releases/tag/v0.10.0) | +| RPC Endpoint | | +| GRPC Endpoint | | +| REST Endpoint | | +| Block Explorer | [https://explore-dorado.fetch.ai/](https://explore-dorado.fetch.ai/) | +| Ledger Explorer | [https://browse-dorado.fetch.ai/](https://browse-dorado.fetch.ai/) | +| Token Faucet | Use block explorer | +| Genesis | `curl https://storage.googleapis.com/fetch-ai-testnet-genesis/genesis-dorado-827201.json --output ~/.fetchd/config/genesis.json` | +| Seed Node(s) | eb9b9717975b49a57e62ea93aa4480e091ae0660@connect-dorado.fetch.ai:36556,46d2f86a255ece3daf244e2ca11d5be0f16cb633@connect-dorado.fetch.ai:36557,066fc564979b1f3173615f101b62448ac7e00eb1@connect-dorado.fetch.ai:36558 | +| Snapshots |

| diff --git a/docs/docs/snapshots.md b/docs/docs/snapshots.md new file mode 100644 index 00000000..c1c9e8dd --- /dev/null +++ b/docs/docs/snapshots.md @@ -0,0 +1,52 @@ +# Chain State Snapshots + +As blockchains get longer, the process of syncing from the genesis block begins to take many hours, or even days to complete. +In circumstances where a faster sync is required, various snapshots of the fetchd chain state data are available for download, to more quickly bootstrap a node. + +Snapshots are available for both mainnet and the most recent testnet. The URLs can be obtained from the [network page](../networks/). +We aim to update snapshots on a daily basis. + +The example below uses the pruned mainnet snapshot, but can be adapted as required for full or archive nodes. + +## Using a snapshot + +### Stop your node + +If you are already running fetchd, it is important that you stop it before proceeding. Instructions for this are highly installation dependent and beyond the scope of this document, but could be as simple as a Ctrl-C. +If you have not already initialised your node, follow the instructions for [joining a testnet](../joining-a-testnet/) (modifying for mainnet as appropriate), then return to this page before starting fetchd. + +### Reset your node + +WARNING: This will irreversibly erase your node's state database. Ensure you take whatever backups you deem appropriate before proceeding. + +If using fetchd <= 0.10.3 +`fetchd unsafe-reset-all` + +If using fetchd >= 0.10.4 +`fetchd tendermint reset-state` + +### Download and install the snapshot + +Many options here! The example below assumes a bash-like environment, uses a single connection for downloading, confirms the md5sum of the downloaded data against that of the original, and does not land the original compressed data to disk. This is a good starting point, but depending on your local environment you may wish to make adaptations that eg sacrifice disk space and extra md5sum complexity for the benefit of parallel downloads with aria2. Entirely up to you! + +```bash +# (optional) show the timestamp of the latest available snapshot +echo "Latest available snapshot timestamp : $(curl -s -I https://storage.googleapis.com/fetch-ai-mainnet-snapshots/fetchhub-4-pruned.tgz | grep last-modified | cut -f3- -d' ')" + +# download, decompress and extract state database +curl -v https://storage.googleapis.com/fetch-ai-mainnet-snapshots/fetchhub-4-pruned.tgz -o- 2>headers.out | tee >(md5sum > md5sum.out) | gunzip -c | tar -xvf - --directory=~/.fetchd + +# (optional, but recommended) compare source md5 checksum provided in the headers by google, with the one calculated locally +[[ $(awk -F\" '/etag:/{ print $2 }' headers.out) == $(awk '{ print $1 }' md5sum.out) ]] && echo "OK - md5sum match" || echo "ERROR - md5sum MISMATCH" + +# (optional) show the creation date of the downloaded snapshot +echo "Downloaded snapshot timestamp: $(grep last-modified headers.out | cut -f3- -d' ')" +``` + +### Restart your node + +Again, this entirely depends on your local installation, but a simple example for mainnet might be... + +```bash +fetchd start --p2p.seeds 17693da418c15c95d629994a320e2c4f51a8069b@connect-fetchhub.fetch.ai:36456,a575c681c2861fe945f77cb3aba0357da294f1f2@connect-fetchhub.fetch.ai:36457,d7cda986c9f59ab9e05058a803c3d0300d15d8da@connect-fetchhub.fetch.ai:36458`. +``` From 3fb56ce1ae7126bfba08e8be5a1309ea87441604 Mon Sep 17 00:00:00 2001 From: Ian Harris Date: Thu, 28 Apr 2022 11:20:03 +0100 Subject: [PATCH 2/4] docs: add snapshot instructions --- docs/mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index bcff822a..d873b7c2 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -16,6 +16,7 @@ nav: - Installation: 'building.md' - Versions: 'versions.md' - Networks: 'networks.md' + - Snapshots: 'snapshots.md' - Commandline Client: - Introduction: 'cli-introduction.md' - Keys: 'cli-keys.md' From df97e1741db58e98d8a1ec601bc6c3dd503a12fe Mon Sep 17 00:00:00 2001 From: Ian Harris Date: Thu, 28 Apr 2022 15:22:42 +0100 Subject: [PATCH 3/4] fix: use x-goog-hash rather than etag --- docs/docs/snapshots.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docs/snapshots.md b/docs/docs/snapshots.md index c1c9e8dd..27a95016 100644 --- a/docs/docs/snapshots.md +++ b/docs/docs/snapshots.md @@ -27,7 +27,7 @@ If using fetchd >= 0.10.4 ### Download and install the snapshot -Many options here! The example below assumes a bash-like environment, uses a single connection for downloading, confirms the md5sum of the downloaded data against that of the original, and does not land the original compressed data to disk. This is a good starting point, but depending on your local environment you may wish to make adaptations that eg sacrifice disk space and extra md5sum complexity for the benefit of parallel downloads with aria2. Entirely up to you! +Many options here! The example below assumes a bash-like environment, uses a single connection for downloading, confirms the md5sum of the downloaded data against that of the original, and does not land the original compressed data to disk. This is a good starting point, but depending on your local environment you may wish to make adaptations that eg sacrifice disk space and extra md5sum complexity for the benefit of parallel downloads with aria2. Entirely up to you... let us know how you get on! ```bash # (optional) show the timestamp of the latest available snapshot @@ -37,7 +37,7 @@ echo "Latest available snapshot timestamp : $(curl -s -I https://storage.google curl -v https://storage.googleapis.com/fetch-ai-mainnet-snapshots/fetchhub-4-pruned.tgz -o- 2>headers.out | tee >(md5sum > md5sum.out) | gunzip -c | tar -xvf - --directory=~/.fetchd # (optional, but recommended) compare source md5 checksum provided in the headers by google, with the one calculated locally -[[ $(awk -F\" '/etag:/{ print $2 }' headers.out) == $(awk '{ print $1 }' md5sum.out) ]] && echo "OK - md5sum match" || echo "ERROR - md5sum MISMATCH" +[[ $(grep 'x-goog-hash: md5' headers.out | sed -z 's/^.*md5=\(.*\)/\1/g' | tr -d '\r' | base64 -d | od -An -vtx1 | tr -d ' \n') == $(awk '{ print $1 }' md5sum.out) ]] && echo "OK - md5sum match" || echo "ERROR - md5sum MISMATCH" # (optional) show the creation date of the downloaded snapshot echo "Downloaded snapshot timestamp: $(grep last-modified headers.out | cut -f3- -d' ')" From b46ba52d94fc8f03c292311004480ff171b2d7a1 Mon Sep 17 00:00:00 2001 From: Ian Harris Date: Thu, 28 Apr 2022 15:23:54 +0100 Subject: [PATCH 4/4] fix: specify 0.10.3 --- docs/docs/networks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/networks.md b/docs/docs/networks.md index 0843ebea..b3ab2c88 100644 --- a/docs/docs/networks.md +++ b/docs/docs/networks.md @@ -11,7 +11,7 @@ The chain identifier of our production network is fetchhub-4. | Date range | 05/04/2022 --> | | Denomination | afet | | Decimals | 18 (1fet = 1000000000000000000afet) | -| Version | [v0.10.2](https://github.com/fetchai/fetchd/releases/tag/v0.10.2) | +| Version | [v0.10.3](https://github.com/fetchai/fetchd/releases/tag/v0.10.3) | | RPC Endpoint | | | GRPC Endpoint | | | REST Endpoint | |