Antelope Spring v1.0 Upgrade Guide

by

On Wednesday, September 25, block producers (BPs) on the EOS blockchain will execute a hard fork to the Spring v1.0 blockchain software, having upgraded their nodes and engaged in a go/no-go vote. 

This new release includes the new Savanna consensus algorithm, which BPs will activate the same day, September 25. 

It also introduces BLS finalizer keys, which all BPs must generate with spring-util and, after September 25, register with the eosio regfinkey action.

Upgrade Steps

All node operators must complete the following steps before September 25:

  1. Generate a snapshot
  2. Stop nodeos
  3. Remove shared_memory.bin 
  4. Upgrade node software to Spring v1.0.0 
  5. Remove the producer-threads option in each node configuration
  6. Generate finalizer key(s) using spring-util
    • Configure finalizer keys. Each individual nodeos instance must have a new and unique set of finalizer keys defined using signature-provider in config.ini or in the command line.
  7. Restart the node from the snapshot

On September 25, the network is expected to sign a multisig transaction to activate BLS protocol features, allowing block producers to register their finalizer keys.

  1. After BLS protocol feature activation, register the first finalizer key using the regfinkey action in the eosio contract
    • The first time it is performed, this action will also activate the finalizer key. Additional finalizer keys can be registered with regfinkey, and then later activated using the actfinkey action in the eosio contract. 

After these steps, the nodes are ready for Savanna consensus activation.

  • Important note: improper finalizer key handling can lead to syncing issues with the network. You can find more details below, or in the official documentation. 
  • Nodes using State History will need to utilize an alternative means of log file compression, such as ZFS LZ4 compression, as SHiP log compression has been removed. However, SHiP log files from Leap 4 and above will continue to work with Spring 1.0, and do not need to be regenerated.
  • Private networks will also need to upgrade system contracts to EOS System Contracts v3.6.0. Public networks will upgrade system contracts with a multisig transaction. 
  • Additionally, top BPs will need to upgrade the Contract Development Toolkit (CDT) in order to build EOS System Contracts v3.6.0 from source. Anyone building BLS-based smart contracts (like the System Contracts v3.6.0) from source will need to upgrade to CDT v4.1.0.
  • The block log, state history logs, and trace logs from prior versions all remain compatible with Spring v1.

Savanna Activation

On September 25, block producers will perform a go/no-go vote to validate 15 of 21 finalizer keys. If the vote is a go, then BPs will sign a multisignature transaction to execute the switchtosvnn action, activating the Savanna consensus algorithm on the EOS Network. 

Spring v1.0 and the Savanna consensus algorithm represent a leap forward in blockchain technology. Savanna uses an upgraded validator architecture that allows Antelope networks to expand the set of active block producers without increasing time to finality. It also reduces the time to finality by over 95%, which is useful for all transactions. This speed profoundly transforms the utility of Antelope’s IBC protocol into a fully horizontally scalable network of networks. 

Important Release Notes

Most new features of Spring 1.0 are related to the Savanna consensus algorithm, although some components have been removed or modified:

  • Node operators must remove the producer-threads configuration option from config.ini, as the option has been removed from the software. The nodeos software will NOT start if producer-threads is included in configuration options.
  • State history (SHiP) log file compression has been disabled. Node operators with state history will need to implement their own compression solutions. Some block producers currently use ZFS LZ4 compression for this purpose.

Spring 1.0 also includes changes relevant to many block producers and other node operators:

  • The new configuration option vote-threads must be set to a positive number (1 or greater) on all intermediate peer/sentry nodes connected to BP nodes; otherwise, votes will NOT propagate through the network. The default value for vote-threads is 4 for Block Producer nodes, but may be set to 0 for other configurations. You can find an example peering setup from EOS Nation, and learn more about peering in the official documentation.
  • The get_block_header_state endpoint now returns a different response, which contains all fields but with only block_num, id, header, and additional_signatures filled out.
  • For nodes running SHiP to support Inter-Blockchain Communication, the finalizer-data-history option MUST be set to true.
  • The finalizer-dir option allows node operators to optionally set the directory path for storing voting history. This is to avoid node syncing issues for setups that move or delete the data directory.

All Node Operators should take note of these details to prepare for the Spring 1.0 hard fork.

Upgrade Checklist

Some specific steps may vary based on node configuration, but these general steps work for most nodes.

Planning for the upgrade

  • Do not test upgrades on your production node—use a test node first
  • Ensure your Ubuntu build is up to date. Older versions of Ubuntu, such as 18.04, are NOT supported. The supported operating systems are:
    • Ubuntu 22.04 Jammy (Recommended)
    • Ubuntu 20.04 Focal
  • Do NOT use deprecated plugins or options
    • producer-threads
    • SHiP (State History) log file compression (use ZFS instead)
  • Enable any new required plugins and options (see below)
  • Make a backup of your node

Upgrading your node

  1. Download / build binaries.
    1. Spring 1.0.0
      This is necessary for all nodes.
    2. (some nodes) EOS System Contracts v3.6.0
      These are necessary only for private networks. Public networks push out system contract updates via a multi-signature transaction approved by 15/21 top block producers.
    3. (some nodes) CDT v4.1.0
      This update is only needed for building system contract binaries or other contracts that use BLS functions.
  2. Ensure that producer_api_plugin is enabled. This is required for taking a snapshot, and unless this plugin was already being used on the node, it’s recommended to disable it after taking a snapshot.
    1. Pass --plugin eosio::producer_api_plugin on the command line, or plugin = eosio::producer_api_plugin in the config file.
  3. Make a snapshot using the /v1/producer/create_snapshot endpoint.
    1. Use curl -X POST http://127.0.0.1:8888/v1/producer/create_snapshot. Wait until curl returns with a JSON response containing the filename of the newly created snapshot file. 
      • This step requires nodeos to continue receiving blocks from the p2p network, otherwise the curl command will never return.
    2. Alternatively, you can download snapshots from top block producers like EOSphere, EOSUSA, Genereos, and EOS Nation. The snapshot produced from Leap v5.0 will be in v6 format. Spring 1.0 is compatible with v4, v6, and v7 snapshots.
  1. Stop your node.
  2. Remove old state data.
    • The block log, state history logs, and trace logs from prior versions are all compatible with Spring v1. There is no need to remove these files when performing the upgrade procedure.
      1. Remove /data/state/shared_memory.bin. This file will be in $HOME/local/share/eosio/nodeos/data/state, or in the directory set with the nodeos --data-dir argument.
      2. (Optional) remove the protocol_features directory.
  1. Remove old configuration options and plugins and add any new ones.

    Remove:
  • The producer-threads option.
  • Some plugins or options were removed or deprecated in Leap v5.0.0. Ensure nodes are not using any of the following:
    • cpu-effort-percent
    • last-block-cpu-effort-percent
    • last-block-time-offset-us
    • produce-time-offset-us
    • max-nonprivileged-inline-action-size
    • max-scheduled-transaction-time-per-block-ms
    • disable-subjective-billing
    • Actions that utilize deferred transactions

Find out more info on disabling these features in the 5.0 Upgrade Guide.

  1. Update your binaries.

Optionally, depending on your preference and setup:

  • Add finalizers-dir to set a directory path for storing voting history. Most nodes can leave this as the default. 
    • Scripts that move or delete the ‘data’ directory need to protect the finalizer safety file, or utilize this option to set another location for the finalizer safety.dat file.
  • Add finality-data-history for nodes running SHiP to support IBC.
  • Prepare to handle new responses to get_block_header_state API calls.
  • Nodes with state history turned on should use an alternate compression solution such as ZFS-LZ4 for SHiP (State History) log file compression. Find more details about ZFS compression on Antelope nodes here.
  • vote-threads should generally be left unchanged at its default. BP nodes have this default set at 4, while other node types may have this default set at 0.  The default values generally work, but intermediate nodes between BP nodes and peer nodes may need to set this value at 1 or greater. The intermediate node must have at least one vote thread in order to properly broadcast votes to the rest of the network.
  1. Generate multiple unique finalizer keys using spring-util and add to signature-provider in config.ini. More details below.
  2. Start your node.

Once a node has been updated to Spring 1.0, it can generate finalizer keys for later use. However, before finalizer key activation, the network must enable the BLS_PRIMITIVES2 protocol features, which will require a multisig transaction scheduled for September 25. Once BLS primitives are activated, you can register and activate finalizer keys. 

Finalizer Key Safety

Note that each block production node must have its own new and unique finalizer key. This means if you have a primary BP node and a backup BP node, each one needs a list of unique finalizer keys, with no sharing keys between nodes. Reusing finalizer keys between nodes will desynchronize the producer with the rest of the chain. 

An important note about the safety.dat file: this file keeps a record of voting history for a given BLS finalizer key. Any mismatch in finalizer key voting history will result in the block producer voting on the wrong branch of the chain and losing sync with the network. This has several implications:

  • DO NOT share BLS finalizer keys between hosts. Instead, activate a new key when switching from one active host to the other.
  • DO NOT share the safety.dat file between hosts or block producers. 
  • DO NOT backup and restore the safety.dat file. Instead, switch to a new BLS finalizer key.

These considerations are important because votes are submitted against a Merkle Tree Digest that is built incrementally off the previous Digest. Using an old voting history will use old Merkle Tree Digests to calculate a new digest, which will be incorrect. Once an incorrect digest is part of the calculation, all the future digests will be incorrect. 

This is why best practice is to never reuse BLS finalizer keys, and always activate a new, unique finalizer key when switching nodes.

Switching to a new key starts fresh with no history. Starting with no history creates the Merkle Tree Digest from the existing information, preventing mismatches. 

Find out more about finalizer safety and managing finalizer keys in the official documentation, or see an example finalizer key rotation.

Finalizer Key Details

To generate and activate your finalizer keys, follow these steps:

  1. Generate your key(s) using spring-util.

Example:

spring-util bls create key --to-console > producer-name.finalizer.key

This will generate a Private key, a Public key, and a proof_of_possession. 

Response:

Private key: PVT_BLS_9-9zi*******oiz-ZZz**************9BZ-o9*******ZU
Public key: PUB_BLS_SvL************ZZ-Zez****9Zb-Z9z****************************************iZZ9P_kzZ*****ezL-Z9z**************************zlZu-Gs*****9Z
Proof of Possession: SIG_BLS_ZPZ********************************99Z-9rZ******************************************************9zZrU-ZZu************ZbZ_yZ****Z-yZP****************************************ZZZzTtVZbcZ-Zck_ZZU********************************************VZ9zZZt_ZlZ************Z9-Z

  1. Add the signature-provider option and one or more keys in config.ini.

First, add one or more of your generated keys to the signature-provider configuration option in the config.ini file of one nodeos instance. 

The format of the option is signature-provider = <PUBLIC_KEY>=KEY:<PRIVATE_KEY> with additional keys on new lines. You may have to add the option if it is not already in the file. 

The keys listed here should be different between nodes, and any key you plan to register and activate must be listed here before booting the node. 

Example:

signature-provider = PUB_BLS_SvL**********zZZZ-Zezlrst9Zb-Z9zZ****************************************ZZ9P_kz******ezL-Z9zZ9zzZb9Z********************lZu-Gs*****9Z=KEY:PVT_BLS_9-9z*********iz-ZZz************aS9BZ-o9*******ZU

signature-provider = PUB_BLS_Zez****9Zb-Zaz*********************************ZZ-9UV*********************Uzf_GLt******Zz-vSZ9**************************zlZu-GZ****ZnZ=KEY:PVT_BLS_9-Zc********Zzz-Zo***************9u9-Zf*******nU

  1. Restart nodeos with the new configuration.
  2. Register a key on-chain with the regfinkey action.

This action registers the key as a potentially active key. The first time a producer calls this action, the key is automatically activated. 

It is recommended to register multiple finalizer keys, although each producer node can only activate one finalizer key at a time.

Example:

cleos push action eosio regfinkey '{"finalizer_name":"NewBlockProducer", \
"finalizer_key":"PUB_BLS_SvL************ZZ-Zez****9Zb-Z9z****************************************iZZ9P_kzZ*****ezL-Z9z*************************cZzlZu-Gs*****9Z", \
"proof_of_possession":"SIG_BLS_ZPZ******************************9V99Z-9rZ*******************************************************zZrU-ZZu************ZbZ_yZ***ZZ-yZP*********************************************tVZbcZ-Zck_ZZ**************************************************Zt_ZlZ**********a9Z9-Z"}' \
-p NewBlockProducer

  1. When switching to a new node, or whenever otherwise desired, activate a new pre-registered key on-chain with the actfinkey action.

After registration, a producer can safely activate any new and unique finalizer key at any time with the actfinkey action. The key used must already exist in the configuration file under signature-provider, and it must already be registered with the regfinkey action.

Example:

cleos push action eosio actfinkey '{"finalizer_name":"NewBlockProducer", \
"finalizer_key":"PUB_BLS_SvL***********ZZZ-Zez****9Zb-Z9z****************************************iZZ9P_kzZ*****ezL-Z9z*************************cZzlZu-Gs*****9Z"}' \
-p NewBlockProducer

  1. If necessary, you can remove a finalizer key with the delfinkey action.

The finalizer_key to be deleted must be a registered key in base64url format, and must not be active. The account removing the finalizer key must have the authority specified in the finalizer_name field.

The delfinkey option will successfully delete an active key if it was the most recently registered finalizer key. If deleting the last registered key, the BP will be removed from the top 21 actively validating BPs, and the next BP in line that has a registered finalizer key will step in.

Example:

cleos push action eosio delfinkey '{"finalizer_name":"NewBlockProducer", \
"finalizer_key":"PUB_BLS_SvL************ZZ-Ze*****9Zb-Z9z****************************************iZZ9P_kz*****tezL-Z9******************************Zu-GsZnZ9I9Z"}' \
-p NewBlockProducer

After registering and activating a new and unique finalizer key, with all nodes upgraded to the Spring 1.0 software, your infrastructure should be ready for the upgrade.

Make Every Node Savanna-Ready

The Savanna consensus algorithm is a leap forward in speed and decentralization for Antelope networks. Upgrading to Spring 1.0 and switching to Savanna involves a series of steps that will ensure a smooth transition. 

BPs and others building system contracts from source should build & upgrade the Contract Development Toolkit to CDT v4.1.0. 

Top block producers on the EOS network must complete the following steps by September 25: 

  1. Build & upgrade node software to Spring v1.0.0
    • Snapshot
    • Stop node
    • Remove old state file shared_memory.bin
    • Remove producer-threads from config.ini 
    • Build and update binaries from source 
    • Restart node
  2. Generate finalizer key(s) using spring-utils.
    • Every nodeos instance should have a set of one or more unique finalizer keys, which are not present on any other nodeos instance. Reusing finalizer keys will lead to the node voting on the wrong branch of the chain.
  3. After the system contract upgrade on September 25, register a finalizer key using the regfinkey action in the eosio contract.
    • The first time a block producer registers a finalizer key, this action will also activate the finalizer key. 
    • Register additional finalizer keys with regfinkey. You must register a key before activating it.
    • Activate registered keys using the actfinkey action.  

By following these steps, block producers can ensure a seamless transition to Savanna, and help unlock new capabilities of the EOS blockchain. For more information about finalizer keys, managing finalizer keys, switching to Savanna consensus, and upgrading to Spring 1.0, check out the migration guides in the official documentation. Upgrade now to spearhead a more powerful and decentralized EOS Network.

Daniel Keyes

Chief Operating Officer (COO)
Responsibilities include: product management, operations, community
Location: Toronto, Canada

Prior to founding the first EOS community in Toronto and co-founding EOS Nation, Daniel spent a decade in the financial technology industry working several diverse roles. His extensive experience in customer service, sales, sales coaching, agent training, digital marketing, digital process management (lean green belt), and product management (certified scrum master, certified product owner) eventually lead him to consulting for a blockchain dev shop.

Daniel earned a Bachelor of Journalism from Ryerson University in 2009 and worked as a chase producer intern at Global TV.

Daniel lives by the principles of Truth, Love, and Freedom.

Twitter Telegram Envelope Linkedin-in