What is the best way to distribute a task across a dataset that uses a relatively expensive-to-create resource or object for the computation.
# in pandas
df = pd.read_csv(...)
foo = Foo() # expensive initialization.
result = df.apply(lambda x: foo.do(x))
# in dask?
# is it possible to scatter the foo to the workers?
client.scatter(...
I plan on using this with dask_jobqueue with SGECluster.