Celery shared data

Question

I have implemented websocket in Django app using Django-channels, now the front-end send some data through the websocket and i want the current running celery task to be able to read it. I tried creating shared memory static object, but not working.

   SimulationInputs.add(simulation_id=simulation.id, init_data=init_inputs)
    return InteractiveSimulationTask.delay_or_fail(
        simulation_id=simulation.id
    )

class SimulationData:
    data = ''


class SimulationInputs:
    data = None

    @classmethod
    def init_manager(cls, manager):
        manager = Manager()
        cls.data = manager.dict()

    @classmethod
    def add(cls, simulation_id, init_data):
        cls.data[simulation_id] = init_data

    @classmethod
    def write(cls, simulation_id, simulation_data):
        if cls.data.get(simulation_id):
            cls.data[simulation_id] = simulation_data

    @classmethod
    def read(cls, simulation_id, simulation_data):
        simulation_data.data = cls.data.get(simulation_id)

# manage.y
if __name__ == "__main__":
    SimulationInputs.init_manager()

class InteractiveSimulationTask(JobtasticTask):
     def calculate_result(self, simulation_id, **kwargs):
         while True:
           SimulationInputs.read(simulation_id=self.simulation.id, simulation_data=simulation_data)

The task always throw exception cls.data.get(simulation_id): NoneObjectType has no method get

I need to share data between the celery task and the main process.

Any hint?

score 0 · Answer 1 · answered Aug 30 '19 at 18:18

Since you're using celery, you probably have redis or some other memory-store available. Consider using this as your indirection layer, i.e. the read and write methods use the simulation_id as a key to the simulation data

I believe the issue you're facing is due to the lifecycle of the python class. In init_manager when you assign to cls.data you're overwriting the class's property, not the instance's property. This doesn't do what you want it to, as evidenced by the error message: cls.data is going to be None.

What I think you're going for is the "Singleton Pattern". You want to have one and only SimulationInputs object which can read/write the data for each ID. This discussion can help you with implementing a singleton in python

score 0 · Accepted Answer · answered Sep 03 '19 at 12:12

I come up to conclusion that Django and celery should not share the memory, because they are on diff. process and they are diff programs, so they should communicate through socket or messaging system. I solved my problem by using redis Pub/Sub https://redis.io/topics/pubsub.

Celery shared data

2 Answers2