5

If I want to have two Lighthouse seed nodes running on different PCs, where each should know about the other, how should I configure them? Although I have found mention of using multiple Lighthouse instances and seen non-seed node configurations that make use of multiple seed nodes, I have not been able to find an example of multiple seed nodes (specifically Lighthouse) on separate PCs that are aware of one another.

I currently have something like this, but I'm not certain if it is correct.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <configSections>
        <section name="akka" type="Akka.Configuration.Hocon.AkkaConfigurationSection, Akka" />
    </configSections>
    <startup>
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
    </startup>
    <akka>
        <hocon>
            <![CDATA[
                    lighthouse{
                            actorsystem: "MySystem" #POPULATE NAME OF YOUR ACTOR SYSTEM HERE
                        }

                    akka {
                        actor { 
                            provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
                        }

                        remote {
                            log-remote-lifecycle-events = DEBUG
                            helios.tcp {
                                transport-class = "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote"
                                applied-adapters = []
                                transport-protocol = tcp
                                #will be populated with a dynamic host-name at runtime if left uncommented
                                #public-hostname = "POPULATE STATIC IP HERE"
                                public-hostname = "192.168.72.55"
                                hostname = "0.0.0.0"
                                port = 4053
                            }
                        }            

                        loggers = ["Akka.Logger.NLog.NLogLogger,Akka.Logger.NLog"]

                        cluster {
                            #will inject this node as a self-seed node at run-time
                            seed-nodes = [
                "akka.tcp://MySystem@192.168.72.55:4053"
                "akka.tcp://MySystem@192.168.72.56:4053"
              ] #manually populate other seed nodes here, i.e. "akka.tcp://lighthouse@127.0.0.1:4053", "akka.tcp://lighthouse@127.0.0.1:4044"
                            roles = [lighthouse]
                        }
                    }
            ]]>
        </hocon>
    </akka>
    <runtime>
        <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
            <dependentAssembly>
                <assemblyIdentity name="NLog" publicKeyToken="5120e14c03d0593c" culture="neutral" />
                <bindingRedirect oldVersion="0.0.0.0-4.0.0.0" newVersion="4.0.0.0" />
            </dependentAssembly>
        </assemblyBinding>
    </runtime>
</configuration>

With the configuration of the other instance basically the same, but with these differences:

akka.remote:

public-hostname = "192.168.72.56"

akka.cluster:

    seed-nodes = [
  "akka.tcp://MySystem@192.168.72.56:4053"
  "akka.tcp://MySystem@192.168.72.55:4053"
] #manually populate other seed nodes here, i.e. "akka.tcp://lighthouse@127.0.0.1:4053", "akka.tcp://lighthouse@127.0.0.1:4044"

Is this right? Are there things I should change or could improve? What is the best way to ensure everything is working as it should? Should I remove the first seed-node in each of these configurations, since the comments seem to indicate that including it isn't necessary?

If I start the node on 192.168.72.55 and then the node on 192.168.72.56, I see it join on 192.168.72.55 and receive a welcome message on the 192.168.72.56. If I shutdown the 192.168.72.56 instance and start it again, it is welcomed again, but the 192.168.72.55 instance keeps outputting messages such as:

Akka.Cluster.ClusterCoreDaemon: Leader can currently not perform its duties, rea
chability status: [akka.tcp://MySystem@192.168.72.55:4053 -> UniqueAddr
ess: (akka.tcp://MySystem@192.168.72.56:4053, 1180787153): Unreachable
[Unreachable] (2), ], member status: [$akka.tcp://MySystem@192.168.72.55
:4053 $Up seen=$True, $akka.tcp://MySystem@192.168.72.56:4053 $Up seen
=$False]
Akka.Remote.MySystem: Association with remote system akka.tcp:
//MySystem@192.168.72.56:4053 has failed; address is now gated for 5000
 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel leve
l)
   at Akka.Remote.EndpointWriter.Unhandled(Object message)
   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, Parti
alAction`1 partialAction)
   at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorConte
xt.Become>b__0(Object m)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
   at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Akka.Actor.ActorCell.HandleFailed(Failed f)
   at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)]
Akka.Actor.OneForOneStrategy: Disassociated Akka.Remote.EndpointDisassociatedExc
eption: Disassociated
   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel leve
l)
   at Akka.Remote.EndpointWriter.Unhandled(Object message)
   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, Parti
alAction`1 partialAction)
   at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorConte
xt.Become>b__0(Object m)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
   at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Akka.Actor.ActorCell.HandleFailed(Failed f)
   at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)

And later:

Akka.Cluster.ClusterCoreDaemon: Leader can currently not perform its duties, reachability status: [], member status: [$akka.tcp://MySystem@192.168.72.55:4053 $Up seen=$True, $akka.tcp://MySystem@192.168.72.56:4053 $Up seen =$False]
Akka.Remote.ReliableDeliverySupervisor: Association with remote system akka.tcp:
//MySystem@192.168.72.56:4053 has failed; address is now gated for 5000
 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel leve
l)
   at Akka.Remote.EndpointWriter.Unhandled(Object message)
   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, Parti
alAction`1 partialAction)
   at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorConte
xt.Become>b__0(Object m)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
   at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)]
Akka.Actor.OneForOneStrategy: Disassociated Akka.Remote.EndpointDisassociatedExc
eption: Disassociated
   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel leve
l)
   at Akka.Remote.EndpointWriter.Unhandled(Object message)
   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, Parti
alAction`1 partialAction)
   at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorConte
xt.Become>b__0(Object m)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
   at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Akka.Actor.ActorCell.HandleFailed(Failed f)
   at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)

Doing this again I noticed the following messages:

Akka.Cluster.ClusterCoreDaemon: New incarnation of existing member [UniqueAddres
s: (akka.tcp://MySystem@192.168.72.56:4053, 1173825306)] is trying to j
oin. Existing will be removed from the cluster and then new member will be allow
ed to join.
Akka.Cluster.ClusterCoreDaemon: Marking node [akka.tcp://MySystem@192.16
8.72.56:4053] as Down
Akka.Cluster.ClusterCoreDaemon: Leader can perform its duties again

Akka.Cluster.ClusterCoreDaemon: New incarnation of existing member [UniqueAddres
s: (akka.tcp://MySystem@192.168.72.56:4053, 253329532)] is trying to jo
in. Existing will be removed from the cluster and then new member will be allowe
d to join.
Akka.Cluster.ClusterCoreDaemon: Cluster Node [akka.tcp://MySystem@192.16
8.72.55:4053] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://Sup
ervisorSystem@192.168.72.56:4053, status = Down]
Akka.Cluster.ClusterCoreDaemon: Leader is removing unreachable node [akka.tcp://
MySystem@192.168.72.56:4053]
Akka.Event.DummyClassForStringSources: Association to [akka.tcp://MySystem@192.168.72.56:4053] having UID [1180787153] is irrecoverably failed. UID is
now quarantined and all messages to this UID will be delivered to dead letters.
Remote actorsystem must be restarted to recover from this situation.
Akka.Remote.ReliableDeliverySupervisor: Association with remote system akka.tcp:
//MySystem@192.168.72.56:4053 has failed; address is now gated for 5000
 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel leve
l)
   at Akka.Remote.EndpointWriter.Unhandled(Object message)
   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, Parti
alAction`1 partialAction)
   at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorConte
xt.Become>b__0(Object m)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
   at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Akka.Actor.ActorCell.HandleFailed(Failed f)
   at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)]
Akka.Cluster.ClusterCoreDaemon: Node [akka.tcp://MySystem@192.168.72.56
:4053] is JOINING, roles [lighthouse]
Akka.Actor.OneForOneStrategy: Disassociated Akka.Remote.EndpointDisassociatedExc
eption: Disassociated
   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel leve
l)
   at Akka.Remote.EndpointWriter.Unhandled(Object message)
   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, Parti
alAction`1 partialAction)
   at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorConte
xt.Become>b__0(Object m)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
   at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Akka.Actor.ActorCell.HandleFailed(Failed f)
   at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
Akka.Cluster.ClusterCoreDaemon: Leader is moving node [akka.tcp://MySystem@192.168.72.56:4053] to [Up]
foven
  • 658
  • 1
  • 5
  • 15
  • Hi, .... any luck with this ? I'm facing same issue. – Jalal El-Shaer Apr 10 '17 at 13:32
  • Sorry, I don't remember finding a solution to this. In the end, I switched to a completely different approach. You could try asking in https://gitter.im/akkadotnet/akka.net if you haven't already. – foven Apr 10 '17 at 18:57
  • Thanks, I used the public-hostname for the public IP address and kept the hostname="0.0.0.0" to bind to all interfaces. Everything worked perfectly – Jalal El-Shaer Apr 11 '17 at 13:44

1 Answers1

2

Maybe You should use BroadcastRouter to aggregate each nodes. Than each node should tell somethink like "I am" to this router.

In the effect, each BroadcastRouter's child should receive "I am" with Sender property.

In this way, You should use this constructor: https://github.com/akkadotnet/akka.net/blob/master/src/core/Akka/Routing/Broadcast.cs#L266

Paweł Siemienik
  • 54
  • 6