2

I need to communicate update events to all running instances of my python script, and i would like to keep the code as simple as possible. I have zero experience with communicating between running processes. Up until now, i have been reading/writing configuration files, which each instance will read and/or update.

Here is some pseudo code i have written (sort of a simple template) to wrap my head around how to solve this problem. Please try to help me fill in the blanks. And remember, i have no experience with sockets, threads, etc... 

import process # imaginary module

class AppA():
    def __init__(self):
        # Every instance that opens will need to attach
        # itself to the "Instance Manager". If no manager
        # exists, then we need to spawn it. Only one manager
        # will ever exist no matter how many instances are
        # running.
        try:
            hm = process.get_handle(AppA_InstanceManager)
        except NoSuchProgError:
            hm.spawn_instance(AppA_InstanceManager)
        finally:
            hm.register(self)
        self.instance_manager = hm

    def state_update(self):
        # This method won't exist in the real code, however,
        # it emulates internal state changes for the sake of
        # explaination.
        #
        # When any internal state changes happen, we will then
        # propagate the changes outward by calling the
        # appropriate method of "self.instance_manager".
        self.instance_manager.propagate_state()

    def cb_state_update(self):
        # Called from the "Instance Manager" only!
        #
        # This may be as simple as reading a known
        # config file. Or could simply pass data
        # to this method.


class AppA_InstanceManager():
    def __init__(self):
        self.instances = []

    def register_instance(self, instance):
        self.instances.append(instance)

    def unregister_instance(self, instance):
        # nieve example for now.
        self.instances.remove(instance)

    def propagate_state(self):
        for instance in self.instances:
            instance.cb_state_update(data)

if __name__ == '__main__':
    app = AppA()

Any Suggestions?

Joe
  • 21
  • 1
  • When you say "instances", are you launching multiple concurrent interpreters from a shell script or some such, or are you spawning multiple instances from a single python interpreter instance? It makes a difference in terms of what approaches will work. – Silas Ray Apr 16 '12 at 17:02
  • Hmm, not sure exactly (apologies for my ignorance). I believe the first because i can see one instance of "python.exe" (in task manager) for each instance of my script application. Does that help? – Joe Apr 16 '12 at 17:15
  • Yes, it does. I'm pretty sure that precludes using multithread/multiprocess without more significantly rearchitecting your system. I think you have to use an external source like a DB or a file to control inter-process data flow. If you have the flexibility to rearchitect though, mutliprocess/mutlithread may be preferable. – Silas Ray Apr 16 '12 at 17:22

1 Answers1

1

There are a few options for this kind of design.

You could use a message queue, its made for this kind of stuff, e.g. AMQP or some ZeroMQ or something like it.

Or you could use something like Redis or some other (in-memory) database for synchronization.

If you don't want to use something like that, you could use the multiprocessing modules synchronization stuff.

Or use a platform specific IPC system, e.g. shared memory via mmap, sysv sockets, etc.

If you want to do things the way you explained, you could have a look at Twisteds perspective broker.

schlenk
  • 6,556
  • 1
  • 20
  • 27
  • I was about to type out something almost exactly like this. Was going to make a specific point though about choosing between a dedicated server process, or a leader-election type situation like the OPs example is suggesting. Redis would be the dedicated device approach, and a message queue like ZeroMQ would give you the option for either. – jdi Apr 16 '12 at 16:57
  • Redis provides publish/subscribe message queuing. – Marcin Apr 16 '12 at 16:58
  • Python has a Queue module in the stdlib. However. I am wondering if i even need something like a "session manager". If it is possible to find all running instances of an application by "process identifier", or something :/. If so, i could just find all the instances, and then "notify" each instance to go read the new updated config file by passing a message or something. Hmm. I'm just riffing here. I'm a complete idiot of how these communications work. – Joe Apr 16 '12 at 17:02
  • The Queue module doesn't help (well, multiprocessing's queue might), as its for threads and not processes. – schlenk Apr 17 '12 at 19:34
  • If you have a bunch of processes and just need to find all running instances you can of course use system specific methods (e.g. parsing files in /proc or `ps efx` on unix, or the appropriate Win32 APIs on windows). Its the same as the solution above, just that you use the OS as the coordinating instance instead of a database/message queue. – schlenk Apr 17 '12 at 19:38