VersionDB
Last updated
Was this helpful?
Last updated
Was this helpful?
Anyone that has been running archive nodes for a while, is familiar with the issue of the ever growing disk size of their nodes. For Cronos, the IAVL database of archived nodes, the application.db usually grows, but does not shrink. A solution that we have come up with is to replace the current RocksDB with VersionDB.
VersionDB stores multiple versions of on-chain state key-value pairs, without using a merkelized tree structure like IAVL tree, making both DB size and query performance much better than IAVL trees. However, VersionDB does not perform root hash and Merkle proof generation, so for those features we still need the IAVL tree. Grpc queries don't need to support proof generation, so VersionDB alone is enough to support this. Currently the --grpc-only flag for one to start a standalone grpc query service.
After migrating to VersionDB, you can then prune the IAVL tree to reclaim disk size if you don't need to generate Merkle proofs on historical versions.
The limitations of the setup with VersionDB and pruned IAVL tree are:
Currently, this solution is only recommended for archive and non-validator nodes to try (validator nodes are recommended to do pruning).
Different implementations exist for VersionDB. Our current implementation is based on RocksDB's v7's experimental user-defined timestamp, which stores the data in a standalone RocksDB instance. it does not support other db backends yet. The other databases in the node still support multiple backends as before.
Does not support eth_getProof, non-grpc / abci_query for the historical versions that's pruned in IAVL tree. The other APIs should function just like an archive node as before.
Download the archive snapshot from either Quicksync or from the we have provided. After you have extracted the .tar into your $NODE_HOME/data/versiondb folder, you can continue to update config and restart.
To enable VersionDB, you need to add VersionDB to the list of store.streamers in app.toml like this:
The db instance is placed at $NODE_HOME/data/versiondb directory. Currently this path cannot be customized. This will switch the grpc query service's backing store from IAVL tree to VersionDB.
Restart the node. it should start to reindex iavl fastnode now.
We will be following the following workflow:
Extract state change sets from existing archived IAVL tree.
Build the versiondb from the change set files.
Build a clean application.db from the change set files.
Update your binary and stop cronosd.
(Optional) get the end-version before extracting changeset
Extract Change Sets This step takes the longest. The migration process will try to parallelize the tasks as much as possible, and hence will use significant RAM memory. There are however flags for a user to control the concurrency level and RAM usage to adjust on different machine specs. Time taken: around 4 days (512G RAM, r6g.16xlarge) for an archive node
Verify Change Sets Time taken: around 1 hour
Time taken: around 1 hour
Restore the IAVL tree. The restored application.db migration commands don't contain fastnode index
time taken: around x hour
To enable VersionDB, you need to add VersionDB to the list of store.streamers in app.toml like this:
The db instance is placed at $NODE_HOME/data/versiondb directory. Currently this path cannot be customized. This will switch the grpc query service's backing store from IAVL tree to VersionDB.
Restart the node. it should start to reindex iavl fastnode now.
On Cronos, we noticed disk size reductions around ~63% for a Cronos archive node. Of course the reuctions that you will see, depend from case to case, but as you can see there's a good improvement in disk size reduction to be gained when moving to versionDB.
Rocksdb archive node
Versiondb archive node
This tutorial starts from scratch, so prepare enough time to go through this migration, especially the change set extraction may take time in the order of days for it to complete, even with a r6g.16xlarge instance. In case you want to skip this step, we will be supporting a snapshot for this in the near future. For more information on the different steps, see this documentation .
