Horcrux with Namada Blockchain for High Availability and Security

This guide is intended for Namada Validator operators. It outlines the use of Horcrux as a remote signing cluster with three Namada nodes to enhance high availability and security. This approach eliminates single points of failure and enhances validator signing key security.

Design:

• Three Horcrux servers serve as the remote signers cluster.
• Three Namada Nodes are used as validators.

Software Requirements:

• Operating System: Ubuntu 22.04.3
• Application: Horcrux v3.3.2

Hardware Requirements for Signers:

• Three Virtual Private Servers (VPS) with 2 CPUs, 2 GB RAM, and 20 GB SSD each.

Firewall Open Ports:

• Ports 19901 for Nodes and 2222 for Signers (You can choose different ports if desired).

DNS Records:

• Create three DNS A records for nodes: node1, node2, and node3.
• Create three DNS A records for signers: signer1, signer2, and signer3.

--- Please note that you can use IP addresses instead of DNS A records. However, I’m using DNS A records here for simplicity and flexibility in managing domain names and potential future changes. Using a domain name allows easier maintenance if the server’s IP address changes.Using IP addresses directly might provide more security since it can help obscure your setup, making it less easily discoverable via DNS lookups. However, keep in mind that this is not a complete security measure, as IP addresses can still be traced through network scans.

Namada Validator Node Config Requirements :

• tendermint_mode should be set to "Validator"
• For an advanced setup (e.g., adding sentry nodes to enhance protection against DDoS attacks), the usual configuration involves 3 signers, 3 sentry nodes, and 3 validators. In this case: On the sentry nodes, set tendermint_mode = "Full". On the validators, set tendermint_mode = "Validator".
• If not using sentry nodes, simply set tendermint_mode = "Validator".

Run These Steps on All Signer Servers:

1. Create a directory to organize your Horcrux files:

mkdir HorcruxNamada
HORCRUX_PATH=$(pwd)/HorcruxNamada

cd HorcruxNamada

2. Download the horcrux binary v3.3.2

wget https://github.com/strangelove-ventures/horcrux/releases/download/v3.3.2/horcrux_linux-amd64

3. Rename horcrux_linux-amd64 to "horcrux" and copy it to /usr/bin/ and /usr/local/sbin/:

mv horcrux_linux-amd64 horcrux

sudo cp horcrux /usr/bin/

sudo cp horcrux /usr/local/sbin/horcrux

4. Create a horcrux service:

sudo nano /etc/systemd/system/hornamada.service

5. Paste the following content:

[Unit]
Description= horcrux Signer For Namada
After=network.target
 
[Service]
Type=simple
User=YOUR_LINUX_USER
WorkingDirectory=$HORCRUX_PATH #but use the string value
ExecStart=/usr/bin/horcrux start --home $HORCRUX_PATH #but use the string value
Restart=on-failure
RestartSec=3
LimitNOFILE=4096
 
[Install]
WantedBy=multi-user.target

6. Enable the service using:

sudo systemctl enable hornamada.service

Activating the first signer ( Signer 1 ):

1. Copy your Namada priv_validator_key.json from the cometbft/config directory (located in $BASE_DIR) to the newly created HorcruxNamada folder.

(Optional) This guide will be using FQDN instead of IP, port 19901 for Nodes and port 2222 for signers. Feel free to adjust based on your preferences.

2. Run the following commands to initialize the horcrux cluster:

YOURDOMAIN="<YOUR_DOMAIN>"
horcrux config init --node "tcp://node1.$YOURDOMAIN:19901" --node "tcp://node2.$YOURDOMAIN:19901" --node "tcp://node3.$YOURDOMAIN:19901" --cosigner "tcp://signer1.$YOURDOMAIN.net:2222" --cosigner "tcp://signer2.$YOURDOMAIN:2222" --cosigner "tcp://signer3.$YOURDOMAIN:2222" --threshold 2 --grpc-timeout 1500ms --raft-timeout 1500ms --home $HORCRUX_PATH

horcrux create-ecies-shards --shards 3 --home $HORCRUX_PATH

NAMADA_CHAIN_ID="<NAMADA_CHAIN_ID>"
horcrux create-ed25519-shards --chain-id $NAMADA_CHAIN_ID --key-file $HORCRUX_PATH/priv_validator_key.json --threshold 2 --shards 3 --home $HORCRUX_PATH

The above steps will generate cosigner communication encryption keys. Expect to find new files and new folders inside HorcruxNamada:

  priv_validator_key.json
  config.yaml
  cosigner_1/ecies_keys.json
  cosigner_2/ecies_keys.json
  cosigner_3/ecies_keys.json
  state

3. Move your priv_validator_key.json to a secure location, as it will not be needed any more.

4. Create a new file inside the state folder named ${NAMADA_CHAIN_ID}_priv_validator_state.json

This file will hold the signing state for the cluster.

5. Paste the below content into it:

{
  "height": "0",
  "round": "0",
  "step": 3
}

5. Copy the HorcruxNamada directory to the other signers using scp.

After copying your HorcruxNamada folder to the second signer, you will need to delete both folders named cosigner_1 and cosigner_3, along with their content from the HorcruxNamada folder inside signer 2. This will leave only two folders: cosigner_2 and state.

6. Copy cosigner_2 content ( ecies_keys.json ) to the HorcruxNamada Folder:

config.yaml
ecies_keys.json
cosigner_2
state\NAMADA-CHAIN-ID_priv_validator_state.json
state

7. Repeat same steps for signer 3. Thus, the HorcruxNamada folder content in the third server should be:

config.yaml
ecies_keys.json
cosigner_3
state\NAMADA-CHAIN-ID_priv_validator_state.json
state

Apply these steps to the first Namada node

1. Configure Namada to start using the Horcrux cluster for signing blocks by editing the config.toml located in Namada config folder.

Search for

priv_validator_laddr = ""

Replace it with

priv_validator_laddr = "0.0.0.0:19901"

2. Remove the priv_validator_key.json from the node and store it in secure location as we don’t need it anymore

3. Stop the Namada node. ONLY after it stopped, open the file priv_validator_state.json inside the cometbft/data directory and check the “height” number.

4. Go to each signer and edit the NAMADA-CHAIN-ID_priv_validator_state.json file inside the HorcruxNamada/state directory with the “height” number from step 3.

It should now look like this:

{
  "height": "<YOUR_NAMADA_HEIGHT>",
  "round": "0",
  "step": 3
}

5. Start your first horcrux signer process inside Signer_One and check the logs:

sudo systemctl restart hornamada.service && sudo journalctl -u hornamada.service -f --output cat

6. Start the signer 2 and signer 3 horcrux signer processes and watch the logs

7. Start your Namada process on your first node and check the logs.

If everything is working fine, your node should start signing blocks.

8. Install 2 Namada Nodes in different servers and edit their config file as we did with node 1

WARNING :

FOR ALL RUNNING NODES IN THE CLUSTER BE SURE YOU ARE USING priv_validator_laddr = "0.0.0.0:19901" AND REMOVE THE ORIGINAL priv_validator_key.json FROM ALL NODES PLEASE NOTE THAT USING REMOTE SIGNING COULD LEAD TO DOUBLE SIGNING AND SLASHING IF YOUR NODE SIGNED SAME BLOCK TWICE, SO BE SURE THAT NEVER USE LOCAL AND REMOTE SIGNING SAME TIME.

TROUBLESHOOTING :

• check FW ports
• check dns for signers and node DNS A records
• check files and folder paths for horcrux
• check same horcrux version on all signers
• PING RTT time between nodes and signers ( more delay more issues )
• IF you want to change signers IP / DNS A record in horcrux config , you have to recreate it from scratch . changing the signers IP or DNS A record in horcrux config will stop horcrux from working as intended