1

I am using in my app a background job system (Sidekiq) to manage some heavy job that should not block the UI.

I would like to transmit data from the background job to the main thread when the job is finished, e.g. the status of the job or the data done by the job.

At this moment I use Redis as middleware between the main thread and the background jobs. It store data, status,... of the background jobs so the main thread can read what it happens behind.

My question is: is this a good practice to manage data between the scheduled job and the main thread (using Redis or a key-value cache)? There are others procedures? Which is best and why?

damoiser
  • 5,532
  • 2
  • 36
  • 60

2 Answers2

3

Redis pub/sub are thing you are looking for. You just subscribe main thread using subscribe command on channel, in which worker will announce job status using publish command. As you already have Redis inside your environment, you don't need anything else to start.

Sergey Moiseev
  • 2,893
  • 2
  • 23
  • 27
  • +1 thanks to showing me the pub/sub, I didn't know it. Do you think that this is a good practice (using Redis)? Do you know others practices to transmit data between backgrounds jobs and the main thread? – damoiser Mar 18 '14 at 16:23
  • 1
    There is also [pub/sub on top of postgresql](http://ngauthier.com/2013/02/rails-4-sse-notify-listen.html). Before pub/sub I just use server polling. And pub/sub definitely better because it allow you to avoid unnecessary requests. – Sergey Moiseev Mar 18 '14 at 16:33
1

Here are two other options that I have used in the past:

  • Unix sockets. This was extremely fiddly, creating and closing connections was a nuisance, but it does work. Also dealing with cleaning up sockets and interacting with the file system is a bit involved. Would not recommend.
  • Standard RDBMS. This is very easy to implement, and made sense for my use case, since the heavy job was associated with a specific model, so the status of the process could be stored in columns on that table. It also means that you only have one store to worry about in terms of consistency.

I have used memcached aswell, which does the same thing as Redis, here's a discussion comparing their features if you're interested. I found this to work well.

If Redis is working for you then I would stick with it. As far as I can see it is a reasonable solution to this problem. The only things that might cause issues are generating unique keys (probably not that hard), and also making sure that unused cache entries are cleaned up.

Community
  • 1
  • 1
Slicedpan
  • 4,845
  • 2
  • 15
  • 32