`super` in a `typing.NamedTuple` subclass fails in python 3.8

Question

I have code which worked in Python 3.6 and fails in Python 3.8. It seems to boil down to calling super in subclass of typing.NamedTuple, as below:

<ipython-input-2-fea20b0178f3> in <module>
----> 1 class Test(typing.NamedTuple):
      2     a: int
      3     b: float
      4     def __repr__(self):
      5         return super(object, self).__repr__()

RuntimeError: __class__ not set defining 'Test' as <class '__main__.Test'>. Was __classcell__ propagated to type.__new__?

In [3]: class Test(typing.NamedTuple): 
   ...:     a: int 
   ...:     b: float 
   ...:     #def __repr__(self): 
   ...:     #    return super(object, self).__repr__() 
   ...:                                                                         

>>> # works

The purpose of this super(object, self).__repr__ call is to use the standard '<__main__.Test object at 0x7fa109953cf8>' __repr__ instead of printing out all the contents of the tuple elements (which would happen by default). There are some questions on super resulting in similar errors but they:

Refer to the parameter-less version super()
Fail already in Python 3.6 (it worked for me before 3.6 -> 3.8 upgrade)
I fail to understand how to fix this anyway, given that it's not a custom metaclass I have control over but the stdlib-provided typing.NamedTuple.

My question is how can I fix this while maintaining backwards compatibility with Python 3.6 (otherwise I'd just use @dataclasses.dataclass instead of inheriting from typing.NamedTuple)?

A side question is how can this fail at definition time given that the offending super call is inside a method which is not even executed yet. For instance:

In [3]: class Test(typing.NamedTuple): 
   ...:     a: int 
   ...:     b: float 
   ...:     def __repr__(self): 
   ...:         return foo

works (until we actually call the __repr__) even though foo is an undefined reference. Is super magical in that regard?

When I run your code in Python 3.6, I get a representation of the `super` object itself, not a representation of your `Test` instance. — chepner, May 01 '20 at 14:07
Also, it appears that whatever compile-time magic allows for implicit arguments still occurs to do some validation of explicit arguments. — chepner, May 01 '20 at 14:13
using `__repr__ = object.__repr__` in your class definition works for me on Python3.6 & Python3.8 — Azat Ibrakov, May 01 '20 at 14:14
@chepner indeed, now I am starting to get confused about why it worked before. But it did... — Jatentaki, May 01 '20 at 14:16
This issue is caused by the metaclass of `typing.NamedTuple`; `typing.NamedTupleMeta`, that's doing some shenanigans. `super()` requires `__class__` to be available *at compile time*, which isn't the case here, apparently. See also: [Provide `__classcell__` example for Python 3.6 metaclass](https://stackoverflow.com/q/41343263/1016216) — L3viathan, May 01 '20 at 14:16
@AzatIbrakov Thanks, that's a good idea. Please go ahead with posting this as a answer, if nobody answers this more in depth soon, I'll accept it. — Jatentaki, May 01 '20 at 14:17
I second Azat's suggestion: since `object` is necessarily the final class in any class's MRO, you don't need `super`'s "cooperative" powers. — chepner, May 01 '20 at 14:25
`NamedTuple` has a [metaclass](https://github.com/python/cpython/blob/v3.6.3/Lib/typing.py#L2131) that limits overloading magic methods. — pylang, May 04 '20 at 17:10

Azat Ibrakov · Answer 1 · 2020-05-01T14:59:06.107

Unfortunately, I'm not so familiar with CPython internals and classes generation to say why it fails, but there is this CPython bug tracker issue which seems to be related and some words in Python docs

CPython implementation detail: In CPython 3.6 and later, the __class__ cell is passed to the metaclass as a __classcell__ entry in the class namespace. If present, this must be propagated up to the type.__new__ call in order for the class to be initialised correctly. Failing to do so will result in a RuntimeError in Python 3.8.

so probably somewhere during actual namedtuple creation we have a call to type.__new__ without __classcell__ propagated, but I don't know if it's the case.

But this particular case seems to be solvable by not using super() call with explicitly saying that "we need to have __repr__ method of the object class" like

class Test(typing.NamedTuple):
    a: int
    b: float
    __repr__ = object.__repr__

Dimitris Fasarakis Hilliard · Accepted Answer · 2020-05-01T20:18:20.020

I was slightly wrong in the other question (which I just updated). Apparently, this behavior manifests in both cases of super. In hindsight, I should have tested this.

What's happening here is the metaclass NamedTupleMeta indeed doesn't pass __classcell__ over to type.__new__ because it creates a namedtuple on the fly and returns that. Actually, in Python's 3.6 and 3.7 (where this is still a DeprecationWarning), the __classcell__ leaks into the class dictionary since it isn't removed by NamedTupleMeta.__new__.

class Test(NamedTuple):
    a: int
    b: float
    def __repr__(self):
        return super().__repr__()

# isn't removed by NamedTupleMeta
Test.__classcell__
<cell at 0x7f956562f618: type object at 0x5629b8a2a708>

Using object.__repr__ directly as suggested by Azat does the trick.

how can this fail at definition time

The same way the following also fails:

class Foo(metaclass=1): pass

Many checks are performed while the class is being constructed. Among these, is checking if the metaclass has passed the __classcell__ over to type_new.

@Jatentaki The leaking of `__classcell__`, I would've thought so. Supporting `super`, don't know, same way as multiple inheritance [recently changed](https://github.com/python/cpython/pull/19363) to now raise an error. Might still be worth it to create an issue, though. — Dimitris Fasarakis Hilliard, May 01 '20 at 20:52

`super` in a `typing.NamedTuple` subclass fails in python 3.8

2 Answers2