How to pickle a defaultdict which uses a lambda function?

Question

How do I create pickle file of function (defaultDict)? The error I get is can't pickle function objects

from collections import defaultdict
dtree = lambda: defaultdict(tree)

try:    import cPickle as pickle
except: import pickle

#Create defaultdict  object:
hapPkl = dtree()

#Create Pickle file
f  = open("hapP.pkl","wb")
pickle.dump(hapPkl,f)
f.close()

StackTrace:

TypeError                                 Traceback (most recent call last)
<ipython-input-287-376cac3b4f0d> in <module>()
      1 f  = open("hapP.pkl","wb")
----> 2 pickle.dump(hapPkl,f)
      3 f.close()

/usr/lib64/python2.7/copy_reg.pyc in _reduce_ex(self, proto)
     68     else:
     69         if base is self.__class__:
---> 70             raise TypeError, "can't pickle %s objects" % base.__name__
     71         state = base(self)
     72     args = (self.__class__, base, state)

TypeError: can't pickle function objects

In Python 3, pickling functions works if they're defined using a `def` statement, but not if they're defined with a `lambda` (even if the `lambda` is assigned to a variable). Try `def dtree(): return defaultdict(dtree)` maybe? — Blckknght, Jun 20 '16 at 18:38
I hadn't checked when I wrote the comment above, but the behavior is the same in Python 2 (it just has less clear error messages). — Blckknght, Jun 20 '16 at 18:44

score 5 · Answer 1 · answered Jun 20 '16 at 18:43

5

The cPickle error message is a bit misleading; the pickle version is better. It's not that you can't pickle functions; it's that they need to be available by their __name__. A lambda has __name__ set to '<lambda>', so it's not picklable. Define it with def:

def tree():
    return defaultdict(tree)

and it will be picklable. (You'll still need a matching definition of tree available when you unpickle it.)

answered Jun 20 '16 at 18:43

user2357112 supports Monica

215,440
22
321
400

1

Related info can be found here: http://stackoverflow.com/questions/16439301/cant-pickle-defaultdict – Mike Shlanta Jun 20 '16 at 18:44
@user, what do you mean by: matching definition? – Merlin Jun 20 '16 at 18:53
@Merlin: If you try to unpickle this defaultdict in an environment with no definition of `tree`, it won't work. – user2357112 supports Monica Jun 20 '16 at 18:57
@user2357112, what wont work.. the unpickling, or the nesting of defaultdicts. – Merlin Jun 20 '16 at 19:02

martineau · Accepted Answer · 2016-06-20T23:14:29.143

1

A simple workaround would be to implement your tree data-structure differently, without defaultdict:

class DTree(dict):
    def __missing__(self, key):
        value = self[key] = type(self)()
        return value

try:    import cPickle as pickle
except: import pickle

#Create dtree object:
hapPkl = DTree()

#Create Pickle file
f = open("hapP.pkl", "wb")
pickle.dump(hapPkl, f)
f.close()

edited Jun 20 '16 at 23:14

answered Jun 20 '16 at 18:49

martineau

99,260
22
139
249

what would be the benefits of this? – Merlin Jun 20 '16 at 19:15
It's arguably a better way to do it. An important one is that it prints and pretty-prints (`pprint()`) like a normal dictionary. See [_What is the best way to implement nested dictionaries in Python?_](http://stackoverflow.com/questions/635483/what-is-the-best-way-to-implement-nested-dictionaries-in-python/19829714#19829714) – martineau Jun 20 '16 at 23:08

How to pickle a defaultdict which uses a lambda function?

2 Answers2