2

I have a user-defined class 'myclass' that I store on file with the pickle module, but I am having problem unpickling it. I have about 20 distinct instances of the same structure, that I save in distinct files. When I read each file, the code works on some files and not on others, when I get the error:

'module' object has no attribute 'myclass'

I have generated some files today and some other yesterday, and my code only works on the files generated today (I have NOT changed class definition between yesterday and today).

I was wondering if maybe my method is not robust, if I am not doing things as I should do, for example maybe I cannot pickled user-defined class, and if this is introducing some randomness in the process.

Another issue could be that the files that I generated yesterday were generated on a different machine --- because I work on an academic cluster, I have some login nodes and some computing nodes, that differs by architecture. So I generated yesterday files on the computing nodes, and today files on the login nodes, and I am reading everything on the login nodes.

As suggested in some of the comments, I have installed dill and loaded it with import dill as pickle. Now I can read the files from computing nodes to login nodes of the same cluster. But if I try to read the files generated on the computing node of one cluster, on the login node of another cluster I cannot. I get KeyError: 'ClassType' in _load_type(name) in dill.py

Can it be because the python version is different? I have generated the files with python2.7 and I read them with python3.3.

EDIT:

I can read the pickled files, if I use everywhere python 2.7. Sadly, part of my code, written in python 3, is not automatically compatible with python 2.7 :(

simona
  • 1,679
  • 4
  • 25
  • 35
  • 2
    Sounds like the python versions/libraries used might have been different. – Marcus Müller Mar 20 '15 at 16:25
  • Have a look at https://docs.python.org/3/library/pickle.html#data-stream-format - you may need to explicitly set the protocol. – jonrsharpe Mar 20 '15 at 16:26
  • 1
    If the versions of python or the versions of any of the libraries are different, your pickled objects may not translate between machines (seconding @MarcusMüller). When you see `'module' object has no attribute 'myclass'`, that means that `module` is being loaded, but the reference to `myclass` is not being found in it. This often happens when a package moves a function or a class internally, and leaves a name-reference or (worse) an instance of the class to maintain the backward compatibility. The error you are seeing is also common for curried functions and references to class instances. – Mike McKerns Mar 20 '15 at 18:17
  • 1
    You could use a serializer (like `dill`) that has the option of pickling the entire class definition (instead of by reference), and that might insulate you against different versions of python or the library/libraries. It won't do anything for already built pickles however. – Mike McKerns Mar 20 '15 at 18:20
  • setting the protocol does not solve my problem, but now I am using `dill` and it works – simona Mar 23 '15 at 13:53
  • actually, it does work on the same cluster. But, as I have access to two distinct clusters, I have moved the pkl files from cluster A to cluster B, and I cannot read them on cluster B. I get `KeyError: 'ClassType'` in _load_type(name) in dill.py – simona Mar 23 '15 at 15:45
  • The default pickle protocol changed in more recent python 3.x as opposed to python 2.x. In python 3.3, the `DEFAULT_PROTOCOL` is `3`, while in python 2.7 it is `2` -- however, that shouldn't affect the `load`. Your `KeyError` is telling me that the pickle registry can't find a `ClassType`, which is odd (I've never seen that before). Could you post a minimal example either here or on the `dill` issues page on github? (will also follow up here in the latter case) – Mike McKerns Mar 24 '15 at 00:31
  • I will try and produce a minimal example. Now, I can tell you that the class is constituted by 5 distinct dictionaries. Each dictionary contains: 1 dictionary of 6 floats, 1 float, and two other dictionaries, each of them containing several (~10) numpy arrays. – simona Mar 25 '15 at 11:03

1 Answers1

2

Can you from mymodule import myclass? Pickling does not pickle the class, just a reference to it. To load a pickled object python must be able to find the class that was to be used to create the object.

eg.

import pickle

class A(object):
    pass

obj = A()
pickled = pickle.dumps(obj)

_A = A; del A # hide class

try:
    pickle.loads(pickled)
except AttributeError as e:
    print(e)

A = _A # unhide class
print(pickle.loads(pickled))
Dunes
  • 32,114
  • 7
  • 68
  • 83
  • If you see this, you probably actually care about making a class pickleable: https://stackoverflow.com/questions/1939058/simple-example-of-use-of-setstate-and-getstate – borgr Mar 12 '20 at 16:28