Shared Network [Community Bounty available]

Bounties
#1

Status: MVP delivered, 250k NIM still available

Description

To connect to the Nimiq Network, both the Safe and Hub are including an iframe in the page from https://network.nimiq.com. This iframe starts a Nimiq browser node (client) that connects to the network. The Safe and Hub then communicate with this iframe client via RPC to update the users’ balances, listen for and send transactions. Both the Safe and Hub, and any other apps that use the Network iframe, create their own instance of the iframe and thus of the Nimiq client within.

To reduce the amount of Nimiq nodes running at the same time in a browser, and to avoid having to re-establish consensus in each Network iframe, we like to add a method of communication between iframe instances that enables sharing of consensus between multiple users of the Network iframe.

Suggested Solution

A promising idea is to use most browsers’ built-in Broadcast Channel API:

The Broadcast Channel API allows simple communication between browsing contexts (that is windows, tabs, frames, or iframes) with the same origin (usually pages from the same site). - MDN

Because the Network iframes are included from the same URL origin, it is possible for these iframes to communicate directly with each other via a Broadcast Channel.

New iframe instances would detect other, already running instances of the same iframe and would relay all requests to the iframe that already has consensus.

A deterministic means of communication between iframe instances in the channel would have to be developed so that no confusion is created as to which iframe sends and receives broadcast messages.

Resources

Network iframe source: https://github.com/nimiq/network/tree/master/src/v2 (Only v2 of the Network iframe API needs to support this communication channel).

Broadcast Channel API docs: https://developer.mozilla.org/docs/Web/API/Broadcast_Channel_API.

For ideas for cross-window communication and message formats, refer to the Nimiq RPC library: https://github.com/nimiq/rpc.

The RPC library can also be extended to support the use of broadcast channels, however to not increase the size of the RPC library for all other purposes, the channel communication library should maybe extend the RPC classes instead.

Completion Criteria

The proposal is completed when

  • Two websites include the Network iframe, but only one Nimiq node is started and the second iframe uses the consensus of the first iframe.
  • The code is readable and organized so that it can be accepted into the Network repository.
  • If possible, the Network iframe API does not change.

Reward

Total: 1’000’000 NIM

  • 750’000 NIM have be rewarded to @Chugwig for his work on a first implementation that you can find in this PR.
  • 250’000 NIM are still available for code style improvements. If you’re interested in working on them, please see this message and get in touch to coordinate the next steps.
4 Likes
#2

How can we go about testing this? I’ve started my own fork of the repo at https://github.com/MatthewDLudwig/network

#3

Hi!

I have added a demo page for the Network Bounty in the source network repo (linked in the OP). This demo is meant as a starting point for testing network events and a balance-check method. You are welcome to adapt the demo to test more things!

The instructions to run the demo for development are as follows:

  1. Run yarn && yarn build or npm install && npm run build in both the root and the client folder (because the demo needs a NetworkClient build as well).
  2. Edit nimiq-dist/v2/index.html and remove the hash from the included script src, so that the line looks like this: <script src="network.js"></script>
  3. Start the Rollup bundler in watch mode for automatic rebuilds while developing: yarn rollup -c -w (or npm run rollup -c -w)
  4. Start your dev HTTP server in the root of the repository.
  5. Point your browser to http://localhost:<dev port>/demos/broadcast/ to run the demo
  6. Run the demo in a second tab to test shared networking

@Chugwig You will need to rebase your master branch onto the original master branch (or merge it into yours).

Happy coding!

1 Like
#4

Thanks for your help! Spent the day testing and ironing out the bugs as well as adding new features. I feel that the bounty is complete and look forward to feedback:

Repo:

Demo:
https://nim.drawpad.org/network/demos/broadcast/

#5

I’m very sorry the review takes longer than expected. But it’s not forgotten!
Just clicked the “demo” link and kept duplicating and closing some tabs… it seemed then that connecting to the “other tab’s consensus” stopped working at some point. In got stuck at this:

Screenshot from 2020-02-17 21-44-06
Screenshot from 2020-02-17 21-44-06570×504 11.9 KB

The console log show no errors, seems like it just didn’t get update messages from the “consensus host” (how do we call all this? :smiley:)

I was wondering about the time it took to connect through the other host - and also saw your comment in the code that the performance can be improved. Have you experimented with that? What would be realistic numbers? (3 seconds might be slower than just doing a new pico consensus)

Thanks for your work. Let me know if I can help or you need further feedback. FYI, I also tested it here: https://soundcode.now.sh - and I’d say it’s technically sound. :wink:

#6

If you can find a surefire way to reproduce the issue you’re seeing I can look into it.

As for bringing down the 3 seconds, I experimented a little with it when originally working on it but nothing too serious. All I know is it can come down lower, no idea how low it can go.

#7

Feedback for Chugwig’s implementation and the further work on it is now being tracked in this PR:

1 Like
#8

Hi @Chugwig!

After your PR has been reviewed by Soeren @NimiqSoeren and Sebastian, the conclusion is that three aspects are not yet there:

  • code readability: it’s part of the requirements that the code should follow the standards of other open source projects by Nimiq. And yes, the discussion of “what is good code” is usually quite controversial, but when checking the code style, it’s clearer

  • the implementation as is does not work reliably - opening and closing multiple tabs shows that the consensus is not always shared - Soeren researched the issue a bit and found out that it might be related to underlying code, nevertheless, it would need to checked properly to be fixed, leading to issue number three

  • tests: it would be good as part of the development to write basic tests to assure the code works as expected. But we realize that this has not been specified in the requirements and thus is just a “would have been nice” - and it’s something that we’ll need to do before releasing it

So we see two options:

  • If you’re interested and motivated to tackle these issues, we would be happy and glad to help!

  • alternatively, we can offer you ¾ of the reward and will try to find other devs that are interested in helping out or work on it ourselves.

Of course we’d be most happy about option one. :slight_smile: Please let us know.

1 Like
#9

Looking it all over I think I’d rather take the 3/4. Regarding the 3 aspects, I completely agree with them and expected as much given the amount of time I put in, my comments are:

  • Is there a style guide out there? I know I could infer a lot from the code around me, but I saw no need to do it for the bounty, changes are always going to be needed anyway since you may prefer different names for functions variables even if they were stylistically right. But if you guys do want people in the future to maintain some coding style I think a style guide would go a long way in that front (though I would make it targeted at community commits and bounties, not a style guide suggestion to Nimiq’s dev community). I can’t speak for everyone but I’d at least keep it in mind when doing commits if I were aware of one.

  • I saw Soeren’s report and I asked for more info on this thread. From what little I know about it, it sounds hard to replicate and the situation is one that’d only realistically occur when testing. Not that it should break on edge cases :sweat_smile: but given the scope and ideal of the bounties it’s not something I felt was worth hunting down (though if instructions to reproduce were given I’d take a look even after receiving the reward).

  • As for tests, I didn’t think good tests could be written for this feature without using a tool like puppeteer or maybe headless browsers. I played with the idea of pure JS, opening windows in a tab and having them communicate back, but it’d end up being a bigger (yet similar with broadcast channels) beast than the shared network bounty itself. Feel free to do as you wish with the tests, I look forward to reviewing them to keep in mind for future projects of a similar nature (and as with Soeren’s bug above, if I see a place where I could add something I’ll gladly join in)

#10

Ok, thank you for your contribution, I’ll be in touch with you directly for the NIM. :slight_smile:

Would be great if you can support us with getting the remaining work done and maybe also find the right person to continue your efforts.

#11

Concerning the changes to the main post, why haven’t the criteria been updated? Idk how I feel about someone having to reach out to Team Nimiq in order to pick up and continue this bounty.

Seems like more of the same inability to let Nimiq as a project revolve around the community/ecosystem instead of around Team Nimiq.