Terminate a multi-thread python program

Question

How to make a multi-thread python program response to Ctrl+C key event?

Edit: The code is like this:

import threading
current = 0

class MyThread(threading.Thread):
    def __init__(self, total):
        threading.Thread.__init__(self)
        self.total = total

    def stop(self):
        self._Thread__stop()

    def run(self):
        global current
        while current<self.total:
            lock = threading.Lock()
            lock.acquire()
            current+=1
            lock.release()
            print current

if __name__=='__main__':

    threads = []
    thread_count = 10
    total = 10000
    for i in range(0, thread_count):
        t = MyThread(total)
        t.setDaemon(True)
        threads.append(t)
    for i in range(0, thread_count):
        threads[i].start()

I tried to remove join() on all threads but it still doesn't work. Is it because the lock segment inside each thread's run() procedure?

Edit: The above code is supposed to work but it always interrupted when current variable was in 5,000-6,000 range and through out the errors as below

Exception in thread Thread-4 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "test.py", line 20, in run
<type 'exceptions.TypeError'>: unsupported operand type(s) for +=: 'NoneType' and 'int'
Exception in thread Thread-2 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "test.py", line 22, in run

Answer 1

Make every thread except the main one a daemon (t.daemon = True in 2.6 or better, t.setDaemon(True) in 2.6 or less, for every thread object t before you start it). That way, when the main thread receives the KeyboardInterrupt, if it doesn't catch it or catches it but decided to terminate anyway, the whole process will terminate. See the docs.

edit: having just seen the OP's code (not originally posted) and the claim that "it doesn't work", it appears I have to add...:

Of course, if you want your main thread to stay responsive (e.g. to control-C), don't mire it into blocking calls, such as joining another thread -- especially not totally useless blocking calls, such as joining daemon threads. For example, just change the final loop in the main thread from the current (utterless and damaging):

for i in range(0, thread_count):
    threads[i].join()

to something more sensible like:

while threading.active_count() > 0:
    time.sleep(0.1)

if your main has nothing better to do than either for all threads to terminate on their own, or for a control-C (or other signal) to be received.

Of course, there are many other usable patterns if you'd rather have your threads not terminate abruptly (as daemonic threads may) -- unless they, too, are mired forever in unconditionally-blocking calls, deadlocks, and the like;-).

Answer 2

There're two main ways, one clean and one easy.

The clean way is to catch KeyboardInterrupt in your main thread, and set a flag your background threads can check so they know to exit; here's a simple/slightly-messy version using a global:

exitapp = False
if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        exitapp = True
        raise

def threadCode(...):
    while not exitapp:
        # do work here, watch for exitapp to be True

The messy but easy way is to catch KeyboardInterrupt and call os._exit(), which terminates all threads immediately.

Answer 3

A Worker might be helpful for you:

#!/usr/bin/env python

import sys, time
from threading import *
from collections import deque

class Worker(object):
    def __init__(self, concurrent=1):
        self.concurrent = concurrent
        self.queue = deque([])
        self.threads = []
        self.keep_interrupt = False

    def _retain_threads(self):
        while len(self.threads) < self.concurrent:
            t = Thread(target=self._run, args=[self])
            t.setDaemon(True)
            t.start()
            self.threads.append(t)


    def _run(self, *args):
        while self.queue and not self.keep_interrupt:
            func, args, kargs = self.queue.popleft()
            func(*args, **kargs)

    def add_task(self, func, *args, **kargs):
        self.queue.append((func, args, kargs))

    def start(self, block=False):
        self._retain_threads()

        if block:
            try:
                while self.threads:
                    self.threads = [t.join(1) or t for t in self.threads if t.isAlive()]
                    if self.queue:
                        self._retain_threads()
            except KeyboardInterrupt:
                self.keep_interrupt = True
                print "alive threads: %d; outstanding tasks: %d" % (len(self.threads), len(self.queue))
                print "terminating..."


# example
print "starting..."
worker = Worker(concurrent=50)

def do_work():
    print "item %d done." % len(items)
    time.sleep(3)

def main():
    for i in xrange(1000):
        worker.add_task(do_work)
    worker.start(True)

main()
print "done."

# to keep shell alive
sys.stdin.readlines()

Answer 4

I would rather go with the code proposed in this blog post:

def main(args):

    threads = []
    for i in range(10):
        t = Worker()
        threads.append(t)
        t.start()

    while len(threads) > 0:
        try:
            # Join all threads using a timeout so it doesn't block
            # Filter out threads which have been joined or are None
            threads = [t.join(1000) for t in threads if t is not None and t.isAlive()]
        except KeyboardInterrupt:
            print "Ctrl-c received! Sending kill to threads..."
            for t in threads:
                t.kill_received = True

What I have changed is the t.join from t.join(1) to t.join(1000). The actual number of seconds does not matter, unless you specify a timeout number, the main thread will stay responsive to Ctrl+C. The except on KeyboardInterrupt makes the signal handling more explicit.

Answer 5

You can always set your threads to "daemon" threads like:

t.daemon = True
t.start()

And whenever the main thread dies all threads will die with it.

http://www.regexprn.com/2010/05/killing-multithreaded-python-programs.html

Answer 6

If you spawn a Thread like so - myThread = Thread(target = function) - and then do myThread.start(); myThread.join(). When CTRL-C is initiated, the main thread doesn't exit because it is waiting on that blocking myThread.join() call. To fix this, simply put in a timeout on the .join() call. The timeout can be as long as you wish. If you want it to wait indefinitely, just put in a really long timeout, like 99999. It's also good practice to do myThread.daemon = True so all the threads exit when the main thread(non-daemon) exits.

Answer 7

thread1 = threading.Thread(target=your_procedure, args = (arg_1, arg_2))    
try:
      thread1.setDaemon(True)  # very important
      thread1.start()
except (KeyboardInterrupt, SystemExit):
      cleanup_stop_thread()
      sys.exit()

When you want to kill the thread just use:

thread1.join(0)