No MQTT messages in despite the MQTT broker working after network disruption

Here we have one of the possible solutions when MQTT messages are no longer receiver from a running MQTT broker. This particular problem in this guide was so far only encountered after unusal network disruption e.g. a shutdown of the shop floor or an extended hiatus.

Problem

The origin of this problem is after a network problem, the stack might start up on two different nodes, which you can check in UMHLens in the problematic cluster. Check in the cluster under “Nodes” if there is an additional node, which should not be there. It usually is named “k3OS-XXXX, with XXXX being random numbers.

Solution

️Please note that this solution might result in some data loss if data is in the superfluous node.

If that is your issue, please select the superfluous node:

• click on the three dots and the right side of the screen

• select “drain”

  

• wait a couple of minutes until the node is completely drained

• delete the superfluous node

• reboot the stack

• The pods should now again be in the correct pod