IBC Relayers

Relaying on Namada

This section of the documentation describes how to operate a relayer on Namada, how to open IBC connections with other IBC compatible chains and how to test your relayer setup by making a connection between two mock local chains.

Introduction

Relayers are a crucial piece of infrastructure that allow IBC compatible chains to pass messages to one another. A relayer is responsible for posting IBC packets to the destination chain, as well as performing other IBC functions such as creating channels, updating the light client states and checking for misbehavior.

Operating a relayer is permissionless. There must be at least one relayer running on a given channel between two chains if the chains are to pass messages to each other on that channel. IBC transfers that are attempted but not picked up by a relayer will timeout and the tokens will be refunded to the sender.

To operate a relayer, you will need:

  • A dedicated server
  • Hermes; the relayer software. Heliax maintains a fork of Hermes compatible with Namada.
  • Full nodes on both chains; a relayer needs to access several weeks worth of chain history and to frequently post transactions to both chains. For testing, a public RPC node may suffice, but for production it is highly recommended for reliability that relayer operators also operate both full nodes themselves.
  • Gas funds for each chain; since relayers will need to regularly post transactions to both chains, they will need a wallet on each with enough funds to cover gas costs (monitoring and topping up as necessary).

This document covers the essentials for installing, configuring and running a relayer on Namada:

  1. Installing Hermes
  2. Configuring Hermes
  3. Setting up the relayer
  4. Creating an IBC channel
  5. Running the relayer

And for setting up a testing environment by running two local Namada chains, see the section Testing with local Namada chains

Installing Hermes

Heliax maintains a fork of the Hermes relayer software. You can download a prebuilt binary or build it from source.

From binaries

One can download the latest binary release from our releases page (opens in a new tab) by choosing the appropriate architecture.

E.g.

export TAG="v1.8.2-namada-beta11" # or the appropriate release for your version of namada
export ARCH="x86_64-unknown-linux-gnu" # or "aarch64-apple-darwin"
curl -Lo /tmp/hermes.tar.gz https://github.com/heliaxdev/hermes/releases/download/${TAG}/hermes-${TAG}-${ARCH}.tar.gz
tar -xvzf /tmp/hermes.tar.gz -C /usr/local/bin

For some systems, /usr/local/bin is a protected directory. In this case, you may need to run the above command with sudo. I.e

sudo tar -xvzf /tmp/hermes.tar.gz -C /usr/local/bin

This is also true for the command cp ./target/release/hermes /usr/local/bin/ below (see the comment).

From source

export TAG="v1.8.2-namada-beta11" # or the appropriate release for your version of namada
 
git clone https://github.com/heliaxdev/hermes.git
git checkout $TAG
cd hermes
cargo build --release --bin hermes
export HERMES=$(pwd) # if needed

Check the binary:

./target/release/hermes --version

It is recommended to now add hermes to $PATH such that it is callable without any pre-fixes. For ubuntu users, this can be achieved by

sudo cp ./target/release/hermes /usr/local/bin/

Configuring Hermes

Make Hermes config file

The Hermes config.toml file defines which chains and channels the relayer will be responsible for. First, create that file:

export HERMES_CONFIG="$HOME/.hermes/config.toml"
touch $HERMES_CONFIG

When running Hermes commands, if you don't specify the config file path, ~/.hermes/config.toml is read as default.

You can find an example of the config file below. Essentially, you change only the chain IDs, the RPC addresses, and the key names in the config file for Namada. If you don't have nodes, please set up nodes manually or through our scripts.

Example: config.toml
[global]
log_level = 'info'
 
[mode]
 
[mode.clients]
enabled = true
refresh = true
misbehaviour = true
 
[mode.connections]
enabled = false
 
[mode.channels]
enabled = false
 
[mode.packets]
enabled = true
clear_interval = 10
clear_on_start = false
tx_confirmation = true
 
[telemetry]
enabled = false
host = '127.0.0.1'
port = 3001
 
[[chains]]
id = 'shielded-expedition.88f17d1d14'  # set your chain ID
type = 'Namada'
rpc_addr = 'http://127.0.0.1:26657'  # set the IP and the port of the chain
grpc_addr = 'http://127.0.0.1:9090'  # not used for now
event_source = { mode = 'push', url = 'ws://127.0.0.1:27657/websocket', batch_delay = '500ms' }  # set the IP and the port of the chain
account_prefix = ''  # not used
key_name = 'relayer'  # The key is an account name you made
store_prefix = 'ibc'
trusting_period = '4752s'
gas_price = { price = 0.0001, denom = 'tnam1qxvg64psvhwumv3mwrrjfcz0h3t3274hwggyzcee' }  # the price isn't used for now, the denom should be a raw token address retrieve denom by namadaw address find --alias "naan"
rpc_timeout = '30s'
memo_prefix = ''
 
[chains.packet_filter]
policy = 'allow'
list = [
  ['transfer', '<channel-id>'], #channel-id leading to the counterparty channel you want to configure IBC
]
 
[[chains]]
id = 'axelar-testnet-lisbon-3'
type = 'CosmosSdk'
rpc_addr = 'http://127.0.0.1:28657' # RPC of the counterparty chain
grpc_addr = 'http://127.0.0.1:28090' # gRPC of the counterparty chain
event_source = { mode = 'push', url = 'ws://127.0.0.1:28657/websocket', batch_delay = '500ms' }
rpc_timeout = '10s'
account_prefix = 'axelar'
key_name = 'relayer'
store_prefix = 'ibc'
default_gas = 5000000
max_gas = 15000000
gas_price = { price = 0.001, denom = 'uaxl' }
gas_multiplier = 1.1
max_msg_num = 30
max_tx_size = 800000
clock_drift = '5s'
max_block_time = '30s'
trusting_period = '5days' # should be less than unbonding period of the counterparty channel
ccv_consumer_chain = false
sequential_batch_tx = false
memo_prefix = ''
 
[chains.trust_threshold]
numerator = "1"
denominator = "3"
 
[chains.packet_filter]
 policy = 'allow'
 list = [
   ['transfer', '<channel-id>'], # channel id of the Namada side
 ]
 
[chains.packet_filter.min_fees]
 
[chains.address_type]
derivation = "cosmos"
 
# Add all channels you want to process

The path to the config file, which is saved in the variable $HERMES_CONFIG will be useful later.

🧩

Interpreting the toml

Each chain configuration is specified under the [[chains]] object. These are the pieces of this puzzle you want to keep your 👀 on:

  • chains.id is the name of the chain
  • chains.rpc_address specifies the port that the channel is communicating through, and will be the argument for the ledger_address of Namada when interacting with the ledger (will become clearer later)
    • Make sure to change the IP address to the IP address of your local machine that is running this node!
  • chains.grpc_addr currently is not used in Namada but needs to be configured for other Cosmos chains
  • chains.key_name specifies the key of the signer who signs a transaction from the relayer. The key should be generated before starting the relayer and will need to be kept topped up with gas funds.
  • chains.event_source specifies the URL of the chain's websocket. This must be the same as the rpc_address for Hermes to work properly.
  • chains.gas_price.denom specifies the token that the relayer pays for IBC transactions. chains.gas_price.price isn't used for now in Namada.
  • trusting_period specifies the maximum period before which the client can not expire. This should be not more than unbonding time in the particular chain.

You can see more details of configuration in the official Hermes documentation (opens in a new tab).

Export environment variables

You may find it convenient to save certain environment variables. These are:

export CHAIN_A_ID="<replace-with-chain-a-id>"
export CHAIN_B_ID="<replace-with-chain-b-id>"
export HERMES_CONFIG="<replace-with-hermes-config-path>"

Setting up the relayer

Create the relayer account

On each chain, there must be a relayer account. The alias should be the same as chains.key_name in the config. On a Namada chain, this can be done by running

namadaw gen --alias relayer

This will generate a key for the relayer account. The key will be stored in the wallet.toml that is found in the base directory of the node, inside the chain-id folder. For example, if the chain-id is shielded-expedition.88f17d1d14, the wallet.toml will be found in $HOME/.local/share/namada/shielded-expedition.88f17d1d14/wallet.toml (on a Ubuntu machine where base-dir has not been changed from default).

The relayer account should have some balance to pay the fee of transactions. Before creating an IBC channel or relaying an IBC packet, you need to transfer the fee token to the relayer account.

Add the relayer key to Hermes

To sign each transaction, the relayer's key should be added to Hermes with keys add command in advance. It requires the wallet.toml which should have the key of chains.key_name. Once the key has been added, Hermes doesn't need the wallet anymore. (Rather than providing the wallet.toml, you can instead provide a text file which contains the 24 word mnemonic for your key.)

hermes --config $HERMES_CONFIG keys add --chain $CHAIN_ID --key-file $WALLET_PATH
# or
hermes --config $HERMES_CONFIG keys add --chain $CHAIN_ID --mnemonic-file $SEED_PATH

Hermes will store the key in ~/.hermes/keys/${CHAIN_ID} as default. You can specify the directory by setting chains.key_store_folder in the config file.

If you want to use an encrypted key with a password, you have to set an environment variable NAMADA_WALLET_PASSWORD_FILE for the password file or NAMADA_WALLET_PASSWORD to avoid entering the password for each transaction submission.

Verify configuration files

After editing config.toml and adding wallet keys, it's time to test the configurations and ensure the system is healthy. Run the following:

hermes health-check
hermes config validate

If everything was set up correctly, you should see output like:

SUCCESS performed health check for all chains in the config
SUCCESS "configuration is valid"

Creating an IBC channel

All IBC transfers between two chains take place along a given IBC channel associated with a given IBC connection.

Channels must be maintained after creation by regularly submitting headers to keep the client state up to date with the counterparty chain. A client that is allowed to fall out of date may expire, and further IBC transfers along the channel will not be possible until the client is revived through governance. Operators should not create new channels on public networks unneccessarily; instead, they should relay on existing channels if possible.

⚠️

The same asset transferred through different channels will not be fungible on the destination chain. Therefore, to avoid confusion and liquidity fragmentation, it is good etiquette to check with a chain's community before creating any new channels and to use existing channels whenever possible.

If no existing channel can be used, the "create channel" command (below) creates not only the IBC channel but also the necessary IBC client connection.

hermes --config $HERMES_CONFIG \
  create channel \
  --a-chain $CHAIN_A_ID \
  --b-chain $CHAIN_B_ID \
  --a-port transfer \
  --b-port transfer \
  --new-client-connection --yes

Note that the above CHAIN_IDs will depend on your own setup, so do check this for yourself!

When the creation has been completed, you can see the channel IDs. For example, the output text below shows that a channel with ID 955 has been created on Chain A shielded-expedition.88f17d1d14, and a channel with ID 460 has been created on Chain B axelar-testnet-lisbon-3.

SUCCESS Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "shielded-expedition.88f17d1d14",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-2996",
        ),
        connection_id: ConnectionId(
            "connection-1479",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: Some(
            ChannelId(
                "channel-955",
            ),
        ),
        version: None,
    },
    b_side: ChannelSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "axelar-testnet-lisbon-3",
                version: 3,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-884",
        ),
        connection_id: ConnectionId(
            "connection-679",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: Some(
            ChannelId(
                "channel-460",
            ),
        ),
        version: None,
    },
    connection_delay: 0ns,
}

Note the channel id's, as you will need to specify the channel IDs when making a transfer over IBC. You specify channel-955 as a channel ID for a transfer from Chain A to Chain B. Also, you specify channel-460 as a channel ID for a transfer from Chain B to Chain A. (The prefix channel- is always required)

Also note the ClientId for each of the chains; we will need them on the next step:
07-tendermint-2996 for shielded-expedition.88f17d1d14 and 07-tendermint-884 for axelar-testnet-lisbon-3

export CLIENT_A_ID="<replace-with-client-a-id>"
export CLIENT_B_ID="<replace-with-client-b-id>"

Update Hermes configuration

By default, Hermes will attempt to relay packets on all channels between the chains listed in its configuration file. We can restrict Hermes to only relay packets on our newly created channels by updating the Hermes configuration with the following (for each chain):

...
[chains.packet_filter]
policy = 'allow'
list = [
  ['transfer', '<channel-id>'], #channel-id leading to the counterparty channel you want to use for IBC
]
...

Running the relayer

While Hermes is running, it monitors chains via the nodes and relays packets according to monitored events. You will likely want to use systemd, tmux, Docker or some similar solution to ensure Hermes is running persistently in the background.

The command to run Hermes is simply:

hermes --config $HERMES_CONFIG start

You can see more details of Hermes at the official documentation (opens in a new tab).

Updating IBC clients

In order to keep IBC channels open for transfers, they must have active IBC clients. An IBC client on chain A references the state of chain B, and vice versa. These clients must be regularly updated to maintain trust guarantees; if they are allowed to lapse, the client will enter an 'expired' state, and no further IBC transfers will be allowed on that channel until the client is revived through governance.

Whenever Hermes relays an IBC transfer, it will automatically update the clients on both chains. However, since reviving an expired client through governance is a time-consuming task, it is worthwhile to regularly submit manual updates to avoid any issues should transfer frequency be lower than expected.

The "update channel" command (below) manually updates the state of a client on a particular chain.

hermes update client --host-chain $CHAIN_A_ID --client $CLIENT_A_ID
hermes update client --host-chain $CHAIN_B_ID --client $CLIENT_B_ID

It is recommended to schedule the above commands to run periodically (using a scheduler such cron or systemd) so that any client expiration issues can be avoided entirely.

Done!

You're now ready to test transferring assets between the two chains.

Testing with local Namada chains

The script setup-namada in the Heliax Hermes repo will set up two chains, each with one validator node, copy necessary files for Hermes, and make an account for Hermes on each ledger. Also, it will make a Hermes config file config_for_namada.toml in the hermes directory.

First, you will need to export some environment variables:

export NAMADA_DIR="<path-to-namada-source-directory>"
export TAG="v1.8.2-namada-beta11" # or the appropriate hermes version for your namada version
git clone https://github.com/heliaxdev/hermes.git
git checkout $TAG # The branch is the same as our Hermes
cd hermes
./scripts/setup-namada $NAMADA_DIR

In this case, the user doesn't have to wait for sync. If the relayer account on each instance has enough balance, the user can create a channel and start Hermes immediately as explained above. The user finds these chain IDs of the chains in the config file config_for_namada.toml. One can run grep "id" ${HERMES_CONFIG}.

# create a channel
hermes --config $HERMES_CONFIG \
  create channel \
  --a-chain $CHAIN_A_ID \
  --b-chain $CHAIN_B_ID \
  --a-port transfer \
  --b-port transfer \
  --new-client-connection --yes
 
# Run Hermes
hermes --config $HERMES_CONFIG start

Each node's data and configuration files are in hermes/data/namada-*/.namada.

In order to close any ledgers setup by the script, one can run

killall namadan