Albatross Docker Lab Setup

NimiqSoeren · March 27, 2021, 12:22pm

We invite Nimiq’s developer community to test the Albatross Controlled Lab Environment. This setup uses Docker to launch 4 validators in a network, with access to the networking of all four nodes and access to RPC of one node.

Having Rust installed is not required, because the building of the client is also run in Docker.

Setup

Install Docker and be able to run docker and docker-compose commands on the command line. Tutorials on how to set up Docker on your system are available on the internet
Git clone, or download, the main branch albatross of the Albatross repository: https://github.com/nimiq/core-rs-albatross
In the repository’s top folder, run the following command to build the Docker image:
docker buildx build -f build_ubuntu.Dockerfile . -t core --progress=plain
This will download the required Docker images for Rust and compile the Albatross client. This will take a few minutes and use the full CPU. You need at least 3GB RAM to complete the compilation!
To start the lab setup, set the NETWORK_NAME environment variable to the network name you want to use. Example: export NETWORK_NAME=nimiq.local
Run docker-compose up to start the network.

You should see the nodes seed1, seed2, seed3, and seed4 start up, establish connections between them, and then, after a few seconds, start producing blocks.

Stopping, restarting, rebuilding

To stop the network, simply CTRL+C in the terminal where they are running. To reset the state, run docker-compose down.

Whenever you change something in the Rust code, you need to rebuild the Docker image with the docker buildx build ... command from step 3 above and docker-compose down and then docker-compose up the network.

Changing only the docker-compose.yml file does not require a rebuild, but still requires docker-compose down and docker-compose up to relaunch the containers.

Known Issues

The network can most likely not start again from existing state after it was stopped. It works most reliably when starting fresh from the Genesis Block, so when you stopped the network, you need to docker-compose down && docker-compose up to reset and restart.
Syncing up to a running network from an external node does not work reliably yet, the team is actively working on that. For now, you need to make do with the existing validators in the setup (or add additional containers that start from the beginning).

New Issues

https://github.com/nimiq/core-rs-albatross/issues

Before reporting a new issue, check the issue tracker if somebody already reported it.

Report issues and panics with as much detail (debug output, traces) and reproduction instructions to Github at the above link.

RPC

You can connect to the RPC server of seed1. You need to route the hostname seed1.nimiq.local (use whichever NETWORK_NAME you used) to 127.0.0.1 in your /etc/hosts file. This is because Traefik, the reverse proxy that the Docker setup uses, routes requests to containers by hostname, so simply using localhost won’t work.

# In file /etc/hosts
...
127.0.0.1   seed1.nimiq.local

The RPC server is available on port 8648, so point your RPC client to http://seed1.nimiq.local:8648.

RPC Methods

RPC methods are listed on the RPC interfaces:

https://github.com/nimiq/core-rs-albatross/tree/albatross/rpc-interface/src

Method names are converted to camelCase for the RPC server, so get_block_by_number becomes the RPC method getBlockByNumber.

RPC via WebSocket

The RPC server also exposes a WebSocket interface, available at ws://seed1.nimiq.local:8648/ws. The WebSocket server additionally allows you to subscribe to new block hashes via the headSubscribe method (method name may change).

The server accepts messages in both text and binary format - but sends all responses in binary format.

RPC Client

You can use any JSON-RPC 2.0 client.

I am maintaining an Albatross RPC command-line client (https://github.com/sisou/arpl) that you can install with:

npm install -g @sisou/albatross-remote
# For yarn users:
yarn global add @sisou/albatross-remote

Then run arpl help to get an overview of how it works.

The client allows you to import accounts, send and fetch transactions, fetch blocks, start and stop staking and manage validators.

Start a REPL with arpl --host=seed1.nimiq.local repl.

Genesis Account Keys

Private keys and validator secret keys of all accounts and validators of the genesis block are available in the genesis config:

https://github.com/nimiq/core-rs-albatross/blob/albatross/genesis/src/genesis/dev-albatross.toml

Send a Transaction

To send a transaction with the arpl RPC client, these are the commands:

# Start the interactive RPC client
arpl --host=seed1.nimiq.local repl

# Import one of the genesis accounts
> account:import a24591648e20642fe5107d0285c1cc35d67e2033a92566f1217fbd3a14e07abc --unlock
# => Account imported: NQ55 X103 EJXG 3S2U 3JNE L779 560T 3LNE 7S60 (unlocked)

# Send a transaction
> transaction:send \
		"NQ55 X103 EJXG 3S2U 3JNE L779 560T 3LNE 7S60" \ # Sender (your unlocked account)
		"NQ58 VL4Q ALUE CU9N NUEA 1NYU EY6P HDCA 7GUE" \ # Recipient
		1000 # NIM
# => Transaction sent: <tx hash>

# Fetch the transaction
> transaction:get <tx hash>
# => Shows the transaction object

Start Staking

To start staking with the arpl RPC client, these are the commands:

# Start the interactive RPC client
arpl --host=seed1.nimiq.local repl

# If you haven't already, import one of the genesis accounts
> account:import a24591648e20642fe5107d0285c1cc35d67e2033a92566f1217fbd3a14e07abc --unlock
# => Account imported: NQ55 X103 EJXG 3S2U 3JNE L779 560T 3LNE 7S60 (unlocked)

# Send a staking transaction to the staking contract
> stake:start \
		"NQ55 X103 EJXG 3S2U 3JNE L779 560T 3LNE 7S60" \ # Sender (your unlocked account)
		0000000000000000000000000000000000000002 \ # Validator ID of seed2
		500 # NIM
# => Transaction sent: <tx hash>

# Verify that your stake is now listed in the staking contract
> stake:list
# => You will see that your address is now listed with 500 NIM
#    as a staker for the ...002 validator

The Docker setup does not include any pool payout logic, so you will not receive any rewards on your address when you stake (delegate) to a validator! Only the validator reward address receives the block rewards.

Sandman · March 27, 2021, 12:42pm

mario · March 27, 2021, 2:29pm

Very cool! Got it up and running in a few minutes.

@NimiqSoeren You might want to update your post to use a different address for the examples, that one ( NQ32 EGL6 H9C8 0JJB PH4S 7RYY ULRC 5B6N 56RE) actually has all it’s NIM staked so you get “Internal error: Transaction rejected: Invalid”.

Instead you could use NQ55 X103 EJXG 3S2U 3JNE L779 560T 3LNE 7S60 and its corresponding private key which has a bunch of unstaked NIM.

NimiqSoeren · March 27, 2021, 3:04pm

Ups! Thank you Mario, as always, for your keen eye and help! Fixed!

NIM · March 28, 2021, 5:48am

When should someone wanting to start a validator up before the testnet is over start acquiring the supplies needed for running the validator?

mario · March 28, 2021, 8:48am

Today the error when a transaction is invalid, is kind of generic “Transaction rejected: Invalid”. It would be helpful to have a more precise reason, I saw there is already a TODO comment in the code, is it planned? Should I create an GH issue so it can be tracked?

Sandman · March 28, 2021, 9:56am

Been no details on that, I would wager those kind of details will come later when the team gets closer to a true public testnet vs the current lab devnet setup

redmaner · March 28, 2021, 12:20pm

@NimiqSoeren thanks for the info. I have one question. During devnet that was somewhere last year (a long time ago) it was possible to open a websocket connection to 8650 to receive blockchain events. This was super useful. Is this still present in the current state of albatross, or is this replaced by the RPC websocket server? I

NimiqSoeren · March 28, 2021, 12:33pm

Hi! The websocket is now part of the regular RPC server and available under ws://seed1.nimiq.local:8648/ws.

Use the RPC method headSubscribe to start receiving block hashes.

redmaner · March 29, 2021, 5:15pm

Thanks for your reply @NimiqSoeren ! Works fine. If you don’t mind I have two other questions:

I was wondering when block rewards are payed out to a validator? Are rewards distributed when the epoch has ended, or are they already distributed during an epoch?
Is it possible for a delegated staker to stop staking during an epoch, or is a delegated stake set for an entire epoch just like normal validators? As far as I understand when a validator is elected to validate for en epoch, he can only stop validating after that epoch has ended. I was wondering if this also applies to delegated stakes. Or maybe I’m completely wrong…

Bruno · March 30, 2021, 2:40am

The rewards are payed out every macro block for the previous batch (a batch is the chain between any two macro blocks, an epoch is the chain between two election macro blocks, batches are shorter than epochs). So, on the macro block at the end of batch t, we distribute the rewards for batch t-1.
No. To stop staking you need to unlock your stake first, that only happens at the end of an epoch. So, you’re correct, both validators and stakers need to wait until the end of the epoch to stop validating/staking.

redmaner · March 30, 2021, 7:10am

Thanks Bruno, much appreciated as always. I have one final question for now:
Looking at the albatross source code regarding calculating block / batch rewards the supply and timestamp of the genesis block are used. The genesis block is the first block of the blockchain. I was wondering if the genesis block for Nimiq is starting as block 0 or as block 1, and if it is possible to retrieve the genesis block using RPC getBlockByNumber(genesisBlockNumber)?

Reason I’m asking is that we are working on a staking pool implementation, and we would like to have a reliable way to retrieve block rewards / batch rewards. We already opened a feature request for that. In the meantime (since it is not addressed yet) I like to follow the formula that Nimiq albatross is using in my pool implementation. I refer to:

For reference: the feature request regarding rewards via RPC is here: https://github.com/nimiq/core-rs-albatross/issues/176

NimiqSoeren · March 30, 2021, 9:33am

hi!
You can get the genesis block via RPC yes, just fetch block #0. It will tell you the hash, and from that you can match that to a genesis supply that you (for now) cannot get from RPC but must calculate from the genesis config file. Or use the method block_reward_for_batch_with_genesis() and just pass in the genesis block header.

To be able to fetch inherents (the block rewards given out for batches), we need to add another store to index inherents by batch number. @Bruno is that doable?

Bruno · March 30, 2021, 3:39pm

I can add a method to fetch inherents by block number (or batch number), no problem. What would be complicated is a method to fetch inherents by hash, since you can easily have identical inherents on different blocks.

Bruno · March 30, 2021, 3:46pm

Just to add another way, you can get the genesis header by the block number and then input it into genesis_parameters (https://github.com/nimiq/core-rs-albatross/blob/62a31b6ef279ee2698ff3c10dcd2127b3b49e57e/blockchain-albatross/src/reward.rs#L9). That will give you the genesis supply and timestamp.

redmaner · March 31, 2021, 1:40pm

From a pool point of view retrieving the inherents by block number (or batch number) would suffice

redmaner · April 26, 2021, 5:05pm

Hi @NimiqSoeren, I’ve a question regarding the unlock_account RPC method. This method takes duration as a parameter. What is the time unit for this parameter? And is it possible to unlock an account indefinitely, and or possible to call unlock_account consecutive times without errors?

NimiqSoeren · April 27, 2021, 7:51am

It seems the duration parameter is currently not used in the implementation of the function, so at the moment all accounts are unlocked indefinitely. You can pass null to use the default, if this will ever be implemented.

github.com

nimiq/core-rs-albatross/blob/albatross/rpc-server/src/dispatchers/wallet.rs#L97


}


/// # TODO
///
///  - The duration parameter is ignored.
///
async fn unlock_account(
    &mut self,
    address: Address,
    passphrase: Option<String>,
    _duration: Option<u64>,
) -> Result<(), Error> {
    let passphrase = passphrase.unwrap_or_default();
    let account = self
        .wallet_store
        .get(&address, None)
        .ok_or(Error::AccountNotFound(address))?;


    let unlocked_account = account
        .unlock(passphrase.as_bytes())
        .map_err(|_locked| Error::WrongPassphrase)?;

redmaner · April 27, 2021, 5:16pm

Thanks! Totally missed the comment in the code that it is not implemented yet.

NimiqSoeren · July 27, 2021, 3:30pm

Cross-posting from Telegram:

Sebastian:

Hello Devs,

We merged the lab-test commits missing from albatross a couple of days ago. We also fixed a bunch of stuff on albatross such that it is now running a lot smoother and better than before.
We will discontinue the lab-test branch and encourage you guys to start testing on the albatross branch instead.

~~The forum post will be updated accordingly~~ , and the lab test branch will be removed some time in the future. For now we keep it such that we do not break the setups people use for testing without prior notice.

On a personal note I would be very interested in knowing if the timestamp panic still occurs with the albatross code base. I was unable to reproduce it for a while now, but my test setup does differ quite drastically from yours I imagine.

So check it out and let us know how it went for you guys. If you can, please report issues to Github as well.