Reuven Lerner: Why do Python lists let you += a tuple, when you can’t + a tuple?

Let’s say you have a list in Python:

>>> mylist = [10, 20, 30]

You want to add something to that list. The most standard way to do this is with the “append” method, which adds its argument to the end of the list:

>>> mylist.append(40) >>> print(mylist) [10, 20, 30, 40]

But what if you want to add multiple items to a list? If you’re new to Python, then you might think that you can and should use a “for” loop. For example:

>>> mylist = [10, 20, 30] >>> new_items = [40, 50, 60] >>> for one_item in new_items:         mylist.append(one_item) >>> print(mylist) [10, 20, 30, 40, 50, 60]

Great, right? But it turns out that there is a smarter and faster way to do this. You can use the += operator. This operator, which invokes the “iadd” (“inplace add”) method on the object to its left, effectively does what we did above, but in much less code:

>>> mylist = [10, 20, 30] >>> new_items = [40, 50, 60] >>> mylist += new_items >>> print(mylist) [10, 20, 30, 40, 50, 60]

It’s not a huge surprise that += can do this. After all, we normally expect += to add and assign to the variable on its left; it works with numbers and strings, as well as other types. And we know that we can use the + operator on lists, too:

>>> [1, 2, 3] + [4, 5, 6] [1, 2, 3, 4, 5, 6]

Can we join a list and a tuple? Let’s check:

>>> mylist = [10, 20, 30] >>> t = (40, 50, 60) >>> mylist + t Traceback (most recent call last): File "", line 1, in  TypeError: can only concatenate list (not "tuple") to list

In other words: No. Trying to add a list and a tuple, even if we’re not affecting either, results in the above error.

Which is why it’s so surprising to many of my students that the following does work:

>>> mylist = [10, 20, 30] >>> t = (40, 50, 60) >>> mylist += t >>> mylist [10, 20, 30, 40, 50, 60]         

That’s right: Adding a list to a tuple with + doesn’t work. But if we use +=, it does.

What gives?

It’s common, when teaching Python, to say that

x += 5

is basically a rewrite of

x = x + 5

And in the majority of cases, that’s actually true. But it’s not always true.

Consider: When you say “x + y” in Python, the “+” operator is translated into a method call. Behind the scenes, no matter what “x” and “y” are, the expression is translated into:

x.__add__(y)

The “__add__” magic method is what’s invoked on an object when it is added to another object. The object on the right-hand side of the “+” is passed as an argument to the method, while the object on the left-hand side is the recipient of the method call. That’s why, if you want your own objects to handle the “+” operator, you need to define the “__add__” method in your class definition. Do that, and things work just fine.

And thus, when we say “x = x + 5”, this is turned into

x = x.__add__(5)

Meaning: First invoke the method, and then assign it back to the variable “x”. In this case, “x” isn’t changing; rather, the variable is now referencing a new object.

Now consider the “+=” operator: It’s translated by Python into “__iadd__”, short for “inplace add.” Notice the slightly different syntax that we use here:

x += y

is translated into

x.__iadd__(y)

Did you see the difference between __add__ and __iadd__? The latter executes the assignment all by itself, internally. You don’t have to capture its output and assign it back to x.

It turns out that the implementation of list.__iadd__ takes the second (right-hand side) argument and adds it, one element at a time, to the list. It does this internally, so that you don’t need to execute any assignment after. The second argument to “+=” must be iterable; if you say

mylist += 5

you will get an error, saying that integers are not iterable. But if you put a string, list, tuple, or any other iterable type on the right-hand side, “+=” will execute a “for” loop on that object, adding each of its elements, one at a time, to the list.

In other words: When you use + on a list, then the right-hand object must be a list. But when you use +=, then any iterable type is acceptable:

>>> mylist = [10, 20, 30] >>> mylist += [40, 50]       # list >>> mylist [10, 20, 30, 40, 50]  >>> mylist += (60, 70)       # tuple >>> mylist [10, 20, 30, 40, 50, 60, 70]  >>> mylist += 'abc'          # string >>> mylist [10, 20, 30, 40, 50, 60, 70, 'a', 'b', 'c']  >>> mylist += {'x':1, 'y':2, 'z':3}    # dict! >>> mylist [10, 20, 30, 40, 50, 60, 70, 'a', 'b', 'c', 'x', 'y', 'z']

Does this work with other types? Not really. For example:

>>> t = (10, 20, 30) >>> t += [40, 50] Traceback (most recent call last): File "", line 1, in  TypeError: can only concatenate tuple (not "list") to tuple         

What happened here? Let’s check the definition of tuple.__iadd__ to find out:

>>> help(tuple.__iadd__) Traceback (most recent call last):   File "", line 1, in  AttributeError: type object 'tuple' has no attribute '__iadd__'

Wait a second: There is no “__iadd__” method for tuples? If so, then how can “+=” work at all?

Because Python tries to be smart in such cases: If the object implements “__iadd__”, then the “+=” operator invokes it. But if the object lacks an “__iadd__” implementation, then Python does what we all guess it normally does — namely, invoke “__add__”, and then assign the results back to the variable. For example:

>>> class Foo(object):         def __init__(self, x):             self.x = x         def __add__(self, other):             print("In __add__")             return Foo(self.x + other.x)  >>> f1 = Foo(10) >>> f2 = Foo(20) >>> f1 += f2 In __add__ >>> vars(f1) {'x': 30}         

In other words, Python notices that our Foo class lacks an implementation of “__iadd__”, and substitutes “__add__” for it, assigning its result (a new instance of Foo) to the original variable.

But if we add (so to speak) the right method, then it’s invoked:

>>> class Foo(object):         def __init__(self, x):             self.x = x         def __add__(self, other):             print("In __add__")             return Foo(self.x + other.x)         def __iadd__(self, other):             print("In __iadd__")             self.x = self.x + other.x             return self          >>> f1 = Foo(10) >>> f2 = Foo(20) >>> f1 += f2 In __iadd__ >>> vars(f1) {'x': 30}         

In the case of Python lists, __iadd__ was implemented such that it doesn’t just add “other.x” to its own value, but that it iterates over each element of “other.x” in a “for” loop. And thus, while “__add__” with a tuple won’t work, “__iadd__” with just about every iterable data types will.

The post Why do Python lists let you += a tuple, when you can’t + a tuple? appeared first on Reuven Lerner.

Planet Python

Leave a Reply

Your email address will not be published. Required fields are marked *