Simulating network outages

For work, we are running a few crypto miners in a nomad job all connected together in a testnet scenario. The developers of this cryto client would like me to devise a method for temporarily interuptting the connection between selected nodes. It would be preferable to acheive this without interupting the running processes, since the purpose of testing is to see how well the miners can resync after an extended interupt.

Would something like this even be possible in nomad with it’s current feature set? I have full root access to the clients themselves, and I tried just manually setting iptable rules, but even this didn’t seem to work.

Hi @nrdxp :wave:

Simulating network conditions is outside Nomad’s scope, so there isn’t anything built-in.

There are, however, multiple tools that are able to do this. I don’t have a lot of experience with any of them, so I can’t make any recommendations, but hopefully these will point you in the right direction.

  • netem and tc are probably already installed in your machines. You can use these tools to simulate packet loss between Nomad clients and servers for example.
  • iptables, as you mentioned, should work as well so I am curious to hear more about what you tried.
  • ip allows you to manipulate routing rules, so it’s not made for network simulation, but you could use it to filter packets by IP address.
  • mininet is a full network emulator, and by “full” I mean full :grinning_face_with_smiling_eyes:. You can essentially simulate any kind of network condition using Python scripts by changing routing logic. But this also means that it’s a more complex solution, so it can have a steep learning curve. I have a sample script to start a Nomad cluster, but it doesn’t actually do much.

Reading about Nomad’s architecture could be handy if you haven’t checked that yet.

I hope this helps. There are quite a few resources online about Linux network simulation, so there are definitely tools missing from my list.

Keep us updated if you do end up finding a good solution :slightly_smiling_face: