Is it possible to `mock.patch` functions run in multiprocessing (using `spawn`)?

Question

Is it possible to mock.patch a function that will be run in a subprocess, which is created using the multiprocessing module with the start method set to spawn?

If there is no solution to patching a child process which is not forked, what would be the right solution to bypass this problem?

It's important to say that switching to use fork is not a solution to my problem. As of python3.8-macOS, the default behavior for the multiprocessing start method is spawn.

From the multiprocessing documentation:

Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725..

Example code (Run on macos with python >= 3.8):

import multiprocessing
import unittest
from unittest import mock


def baz(i):
    print(i)


def bar(i):  #  Middle function to avoid a pickeling problem with mocks
    baz(i)


def foo():
    with multiprocessing.Pool(2) as pool:
        pool.map(bar, [i for i in range(10)])


class TestClass(unittest.TestCase):
    @mock.patch(f'{__name__}.baz', return_value=None)
    def test_case(self, mock):
        # multiprocessing.set_start_method('fork', force=True)
        foo()

The baz function is not patched in the spawned processes (and hence, it prints) as can be seen in the following output. Changing the default start method (commented in code) solves the problem

Output:

============================================================================== test session starts ===============================================================================
platform darwin -- Python 3.8.0, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
rootdir: /Users/alonmenczer/dev/path_test/proj
collected 1 item

mp_spawn.py 0
1
4
5
6
7
8
9
2
3
.

=============================================================================== 1 passed in 0.27s ================================================================================

Answer 1

I assume you would rather prefer to use mock. But if the library doesn't work, mocking can be achieved directly. E.g.:

def my_mock_baz(func):
    def _mock_baz(i):
        print(42)

    def inner(*args, **kwargs):
        global baz
        backup = baz
        baz = _mock_baz
        res = func(*args, **kwargs)
        baz = backup
        return res

    return inner

And then in the test:

class TestClass(unittest.TestCase):
    @my_mock_baz
    def test_case(self):
        foo()

Of course, I other cases, mocking would be more difficult, but with the flexibility of Python it should be not too hard.

Side note: your problem doesn't reproduce in my Ubuntu 20.04, Python 3.8.2.

Answer 2

yes it definitely is it just depends how

from multiprocessing import Process
import os

def info(title):
    print(title)
    print('module name:', __name__)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())

def f(name):
    info('function f')
    print('hello', name)

if __name__ == '__main__':
    info('main line')
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()