10

Is there any method I can override that will allow me to use print statements / pdb / etc. to keep track of every time an instance of my class is allocated? While unpickling some objects I am seeming to get some that never have either __setstate__ or __init__ called on them. I tried overriding __new__ and printing out the id of every object I make in __new__, but I am still encountering objects with ids that were never printed.

Edit: here is my code I use for altering (instrumenting) __new__ of my class and all of its super-classes except for object itself:

class Allocator:
    def __init__(self, my_class):
       self.my_class = my_class
       self.old_new = my_class.__new__

    def new(self, * args, ** kargs):
        rval = self.old_new(*args, ** kargs)
        #rval = super(self.my_class,cls).__new__(cls)
        print 'Made '+str(self.my_class)+' with id '+str(id(rval))
        return rval

def replace_allocator(cls):
    if cls == object:
        return

    setattr(cls,'__new__',Allocator(cls).new)
    print cls.__base__

    try:
        for parent in cls.__base__:
            replace_allocator(parent)
   except:
        replace_allocator(cls.__base__)

I call replace_allocator on my classes' parent class as soon as it is imported in the main script. My class has a custom __new__ to begin with, which also prints out the id.

Ian Goodfellow
  • 2,384
  • 1
  • 16
  • 19
  • Are you absolutely certain they are different instances? Have you checked if `id(instance)` matches? – Rosh Oxymoron Jan 25 '11 at 20:18
  • Are you trying to use `Allocator(cls)` as a type cast? If so, it is wrong, because in Python it is a constructor call for an `Allocator` object, and the `.new` part will get a new bound-method instance. – Apalala Jan 25 '11 at 23:15
  • Rosh: Yes. My allocator function shows the way that I print out id(instance). Later, when I find an object that is missing some fields that should have been added by setstate or init, I print out its id too. If I grep for that id in the earlier printouts there are no hits. – Ian Goodfellow Jan 25 '11 at 23:49
  • Apalala: It is indeed my intention to use `Allocator(cls)` as a constructor call for an `Allocator` object. The point of an `Allocator` constructed with `cls` is to call the original `__new__` method of `cls` and report the `id` of the resulting object instance. – Ian Goodfellow Jan 25 '11 at 23:51
  • The iteration should be `for parent in cls.__bases__`. You don't need the fall-back to `cls.__base__`. And don't use catch-all `except:` clauses. They always bite you. – Sven Marnach Jan 26 '11 at 12:04
  • Beware that if the class doesn't define `__new__`, `cls.__new__` will be the constructor of its first base. Weird side-effects are to be expected when you replace `__new__` like that. Take it out of `cls.__dict__` instead. You can also simply walk over `cls.mro()` instead of walking the bases yourself. – Rosh Oxymoron Jan 26 '11 at 14:17
  • @Rosh: If a class -- say `A` -- does not define `__new__()`, then calling `A()` will call `A.__new__()` anyway, and the definition of `__new__()` will be taken from some base class. The code above does exactly the same -- it calls `A.__new__()`. There are no weird side effects to be expected. – Sven Marnach Jan 26 '11 at 16:12

3 Answers3

5

(This is more of a comment than an answer.)

Quoting Guido's Unifying types and classes in Python 2.2:

There are situations where a new instance is created without calling __init__ (for example when the instance is loaded from a pickle). There is no way to create a new instance without calling __new__ (although in some cases you can get away with calling a base class's __new__).

If you are using new-style classes (descendants of object), __new__() should always be called. I don't think the obscure cases "you can get away with calling a base class's __new__" in will happen accidently, though I don't know what these cases actually are.

And just to add an example:

In [1]: class A(object):
   ...:     def __new__(cls):    
   ...:         print "A"
   ...:         return object.__new__(cls)
   ...:     

In [2]: A()
A
Out[2]: <__main__.A object at 0xa3a95cc>

In [4]: object.__new__(A)
Out[4]: <__main__.A object at 0xa3a974c>
Sven Marnach
  • 483,142
  • 107
  • 864
  • 776
  • This sounds like a feasible approach but unfortunately if you can get away with calling the base class, then that might explain why there are objects getting allocated without my knowledge. I'm only allowed to override `__new__` for classes other than `object` itself. – Ian Goodfellow Jan 25 '11 at 22:04
0

Are you using new-style classes? For Pickle to call __setstate__, the __getstate__ method should also be defined on the class returning a non-False value.

vz0
  • 30,291
  • 7
  • 37
  • 74
  • This does not answer the question of how to detect objects as they are created. But yes I am using new-style classes and I do have a `__getstate__` implemented. Also, `__setstate__` is getting called on most but not all instances of the class. – Ian Goodfellow Jan 25 '11 at 19:34
0

Not sure if this can help you, but Python's garbage collector has introspective capabilities that might be worth taking a look at.

s.m.
  • 7,427
  • 2
  • 34
  • 44
  • The question is about keeping track of the *creation* of objects. – Sven Marnach Jan 25 '11 at 19:47
  • You're right, I most probably misunderstood the question. But now I wonder, won't the garbage collector need itself to monitor object allocation? If so, one might devise a method to hook into the gc and figure out when an object of interested gets allocated. I'm not questioning what you said, I'm just curious. – s.m. Jan 25 '11 at 20:00