Why does Python use 'magic methods'?

Question

I've been playing around with Python recently, and one thing I'm finding a bit odd is the extensive use of 'magic methods', e.g. to make its length available, an object implements a method, def __len__(self), and then it is called when you write len(obj).

I was just wondering why objects don't simply define a len(self) method and have it called directly as a member of the object, e.g. obj.len()? I'm sure there must be good reasons for Python doing it the way it does, but as a newbie I haven't worked out what they are yet.

Answer 1

AFAIK, len is special in this respect and has historical roots.

Here's a quote from the FAQ:

Why does Python use methods for some functionality (e.g. list.index()) but functions for other (e.g. len(list))?

The major reason is history. Functions were used for those operations that were generic for a group of types and which were intended to work even for objects that didn’t have methods at all (e.g. tuples). It is also convenient to have a function that can readily be applied to an amorphous collection of objects when you use the functional features of Python (map(), apply() et al).

In fact, implementing len(), max(), min() as a built-in function is actually less code than implementing them as methods for each type. One can quibble about individual cases but it’s a part of Python, and it’s too late to make such fundamental changes now. The functions have to remain to avoid massive code breakage.

The other "magical methods" (actually called special method in the Python folklore) make lots of sense, and similar functionality exists in other languages. They're mostly used for code that gets called implicitly when special syntax is used.

For example:

• overloaded operators (exist in C++ and others)
• constructor/destructor
• hooks for accessing attributes
• tools for metaprogramming

and so on...

Answer 2

From the Zen of Python:

In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.

This is one of the reasons - with custom methods, developers would be free to choose a different method name, like getLength(), length(), getlength() or whatsoever. Python enforces strict naming so that the common function len() can be used.

All operations that are common for many types of objects are put into magic methods, like __nonzero__, __len__ or __repr__. They are mostly optional, though.

Operator overloading is also done with magic methods (e.g. __le__), so it makes sense to use them for other common operations, too.

Answer 3

Python uses the word "magic methods", because those methods really performs magic for you program. One of the biggest advantages of using Python's magic methods is that they provide a simple way to make objects behave like built-in types. That means you can avoid ugly, counter-intuitive, and nonstandard ways of performing basic operators.

Consider a following example:

dict1 = {1 : "ABC"}
dict2 = {2 : "EFG"}

dict1 + dict2
Traceback (most recent call last):
  File "python", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'dict' and 'dict'

This gives an error, because the dictionary type doesn't support addition. Now, let's extend dictionary class and add "__add__" magic method:

class AddableDict(dict):

    def __add__(self, otherObj):
        self.update(otherObj)
        return AddableDict(self)


dict1 = AddableDict({1 : "ABC"})
dict2 = AddableDict({2 : "EFG"})

print (dict1 + dict2)

Now, it gives following output.

{1: 'ABC', 2: 'EFG'}

Thus, by adding this method, suddenly magic has happened and the error you were getting earlier, has gone away.

I hope, it makes things clear to you. For more information, refer to:

A Guide to Python's Magic Methods (Rafe Kettler, 2012)

Answer 4

Some of these functions do more than a single method would be able to implement (without abstract methods on a superclass). For instance bool() acts kind of like this:

def bool(obj):
    if hasattr(obj, '__nonzero__'):
        return bool(obj.__nonzero__())
    elif hasattr(obj, '__len__'):
        if obj.__len__():
            return True
        else:
            return False
    return True

You can also be 100% sure that bool() will always return True or False; if you relied on a method you couldn't be entirely sure what you'd get back.

Some other functions that have relatively complicated implementations (more complicated than the underlying magic methods are likely to be) are iter() and cmp(), and all the attribute methods (getattr, setattr and delattr). Things like int also access magic methods when doing coercion (you can implement __int__), but do double duty as types. len(obj) is actually the one case where I don't believe it's ever different from obj.__len__().

Answer 5

They are not really "magic names". It's just the interface an object has to implement to provide a given service. In this sense, they are not more magic than any predefined interface definition you have to reimplement.

Answer 6

While the reason is mostly historic, there are some peculiarities in Python's len that make the use of a function instead of a method appropriate.

Some operations in Python are implemented as methods, for example list.index and dict.append, while others are implemented as callables and magic methods, for example str and iter and reversed. The two groups differ enough so the different approach is justified:

1. They are common.
2. str, int and friends are types. It makes more sense to call the constructor.
3. The implementation differs from the function call. For example, iter might call __getitem__ if __iter__ isn't available, and supports additional arguments that don't fit in a method call. For the same reason it.next() has been changed to next(it) in recent versions of Python - it makes more sense.
4. Some of these are close relatives of operators. There's syntax for calling __iter__ and __next__ - it's called the for loop. For consistency, a function is better. And it makes it better for certain optimisations.
5. Some of the functions are simply way too similar to the rest in some way - repr acts like str does. Having str(x) versus x.repr() would be confusing.
6. Some of them rarely use the actual implementation method, for example isinstance.
7. Some of them are actual operators, getattr(x, 'a') is another way of doing x.a and getattr shares many of the aforementioned qualities.

I personally call the first group method-like and the second group operator-like. It's not a very good distinction, but I hope it helps somehow.

Having said this, len doesn't exactly fit in the second group. It's more close to the operations in the first one, with the only difference that it's way more common than almost any of them. But the only thing that it does is calling __len__, and it's very close to L.index. However, there are some differences. For example, __len__ might be called for the implementation of other features, such as bool, if the method was called len you might break bool(x) with custom len method that does completely different thing.

In short, you have a set of very common features that classes might implement that might be accessed through an operator, through a special function (that usually does more than the implementation, as an operator would), during object construction, and all of them share some common traits. All the rest is a method. And len is somewhat of an exception to that rule.

Answer 7

There is not a lot to add to the above two posts, but all the "magic" functions are not really magic at all. They are part of the __ builtins__ module which is implicitly/automatically imported when the interpreter starts. I.e.:

from __builtins__ import *

happens every time before your program starts.

I always thought it would be more correct if Python only did this for the interactive shell, and required scripts to import the various parts from builtins they needed. Also probably different __ main__ handling would be nice in shells vs interactive. Anyway, check out all the functions, and see what it is like without them:

dir (__builtins__)
...
del __builtins__