asynchronous method calls that return values

Question

I have a polling function that checks a Snowflake database and returns four values:

def __watch_job_status(self, job_id):
        start_time = datetime.datetime.now()
        polling2.poll(
            lambda: self.controller.sf.connector.is_still_running(self.controller.sf.connector.get_query_status_throw_if_error(job_id)) == False,
            step=5,
            poll_forever=True
        )
        polling_time_in_seconds = datetime.datetime.now() - start_time
        shape_str = f"{self.controller.query_ui.container.df.shape[0]} rows, {self.controller.query_ui.container.df.shape[1]} columns"
        return job_id, 'COMPLETE', polling_time_in_seconds, shape_str

Currently I am updating a Jupyter NB UI with the returned results but this is done in a blocking fashion and makes the UI unusable. I have been scouring SO for examples and used this example from @unutbu as my guide.

import multiprocessing as mp

# Inside the class INIT function
self.pool = mp.Pool()

# Inside the function that updates the UI:
# job.id is a string
self.pool.apply_async(self.__watch_job_status, args = (job.id, ), callback = self.__job_results)
self.pool.close()
self.pool.join()

# The callback function
def __job_results(self, result):
    status_df = self.widgets['status_qgrid'].df
    (col_opts, col_defs) = self.__fetch_qgrid_options()
    for i, row in status_df.iterrows():
        if row['ID'] == None:
           row['ID'] = result[0]
           row['STATUS'] = result[1]
           row['TIME'] = result[2]
           row['SHAPE'] = result[3]
    self.widgets['status_qgrid'] = qgrid.show_grid(status_df, column_options=col_opts, column_definitions=col_defs, grid_options={'forceFitColumns': False})
    self.widgets['grid'][0, 0] = self.widgets['status_qgrid']

When I run this code I don't get any errors but the UI is not updating as expected. I expected that the callback function __job_results would get the return value of the Pool process (which called the __watch_job_status function) and then update the UI. Also, I am running on Windows 10. I did notice the example I used for guidance is meant for a script not an application like I have:

if __name__ == '__main__':
    apply_async_with_callback()

thx

EDIT

So this doesn't answer my question about multiprocessing but it does provide a solution. Just call the method that is blocking with a Thread:

t = threading.Thread(target=self.__watch_job_status, args=(job_id,))
t.start()

asynchronous method calls that return values

0 Answers0