41

I am creating a little dashboard for a user that will allow him to run specific jobs. I am using Django so I want him to be able to click a link to start the job and then return the page back to him with a message that the job is running. The results of the job will be emailed to him later.

I believe I am supposed to use subprocess.Popen but I'm not sure of that. So in pseudocode, here is what I want to do:

if job == 1:
    run script in background: /path/to/script.py
    return 'Job is running'
Paul D. Waite
  • 89,393
  • 53
  • 186
  • 261
sheats
  • 29,382
  • 14
  • 41
  • 43
  • possible duplicate of **[Starting a background process in python](http://stackoverflow.com/questions/1196074/starting-a-background-process-in-python)** and of **[How to launch and run external script in background?](http://stackoverflow.com/questions/1605520/how-to-launch-and-run-external-script-in-background)** – oHo Nov 13 '13 at 09:32

4 Answers4

63
p = subprocess.Popen([sys.executable, '/path/to/script.py'], 
                                    stdout=subprocess.PIPE, 
                                    stderr=subprocess.STDOUT)

That will start the subprocess in background. Your script will keep running normally.

Read the documentation here.

nosklo
  • 193,422
  • 54
  • 273
  • 281
  • nosklo: Thanks. How would I pass in arguments to the script? – sheats Feb 13 '09 at 14:09
  • 3
    Additional elements in the list passed as the first argument. The linked documentation is useful, and linked to for a reason. – Devin Jeanpierre Feb 13 '09 at 14:28
  • In a non-windows environment, you could check out os.fork as well – Bodo Thiesen Feb 13 '09 at 14:31
  • Devin: Thanks. I ask on here because it was hard for me to figure that out from the docs. I don't see sys.executable anywhere in the docs. If all we ever needed was docs, we wouldn't need SO =) – sheats Feb 13 '09 at 14:38
  • @sheats: sys.executable is documented here http://docs.python.org/library/sys.html#sys.executable – nosklo Feb 13 '09 at 14:45
  • @nosklo Is this a good thing to do in the context of a web server? I've seen advice to run a separate service for long-running jobs and communicate with a queue mechanism. – akaihola Feb 14 '09 at 11:51
  • @akaihola, it seems to be the original question. sheats is asking for a way to run a separate process (outside the webserver process). A communication mechanism was not part of the question. – nosklo Feb 16 '09 at 13:32
  • stdout=PIPE might hang the subprocess, redirect to DEVNULL to discard the output. – jfs Nov 22 '12 at 16:50
  • @nosklo can we pass a new parameter (just like we pass to a function) from previous process to the new sub-process? – user5319825 Aug 26 '16 at 07:21
6

Running this through a message queue is definitely the way to go if you're thinking about long-term scaling. Send a message to the queue who's running constantly in the background, and write job handlers to deal with the different sorts of messages.

Since you're using Django, I think Beanstalkd is a pretty good fit. Here's a pretty nice tutorial on the subject. The first comment in that article also has some good tips.

Personally I've rolled with a custom in-memory queue server written in Erlang, with Python-bindings written in C. But redis looks like it might work out as a great contender for future queuing/messaging-needs. Hope this helps!

  • 3
    celery (http://pypi.python.org/pypi/celery) would also be a good match for this. It uses RabbitMQ (for the message persistence, a beanstalkd backend for carrot is certainly possible if someone has the time) The beanstalkd tutorial only solves one of the problems, sending messages, but leaves actually executing the task as an exercise for the reader. – asksol Aug 07 '09 at 18:13
3

subprocess.Popen is indeed what you are looking for.

Aaron Maenpaa
  • 105,677
  • 10
  • 91
  • 107
1

Although if you find that you want to start communicating a bunch of information between the subprocess and the parent, you may want to consider a thread, or RPC framework like Twisted.

But most likely those are too heavy for your application.

Sean Cavanagh
  • 3,789
  • 2
  • 17
  • 14