Thomas Guest: Aligning the first line of a triple-quoted string in Python

Python’s triple-quoted strings are a convenient syntax for strings where the contents span multiple lines. Unescaped newlines are allowed in triple-quoted strings. So, rather than write:

song = ("Happy birthday to you\n"         "Happy birthday to you\n"         "Happy birthday dear Gail\n"         "Happy birthday to you\n") 

you can write:

song = """Happy birthday to you Happy birthday to you Happy birthday dear Gail Happy birthday to you """ 

The only downside here is that the first line doesn’t align nicely with the lines which follow. The way around this is to embed a \newline escape sequence, meaning both backslash and newline are ignored.

song = """\ Happy birthday to you Happy birthday to you Happy birthday dear Gail Happy birthday to you """ 

Planet Python

Python Sweetness: Threadless mode in Mitogen 0.3

Mitogen has been explicitly multi-threaded since the design was first conceived. This choice is hard to regret, as it aligns well with the needs of operating systems like Windows, makes background tasks like proxying possible, and allows painless integration with existing programs where the user doesn’t have to care how communication is implemented. Easy blocking APIs simply work as documented from any context, and magical timeouts, file transfers and routing happen in the background without effort.

The story has for the most part played out well, but as work on the Ansible extension revealed, this thread-centric worldview is more than somewhat idealized, and scenarios exist where background threads are not only problematic, but a serious hazard that works against us.

For that reason a new operating mode will hopefully soon be included, one where relatively minor structural restrictions are traded for no background thread at all. This article documents the reasoning behind threadless mode, and a strange set of circumstances that allow such a major feature to be supported with the same blocking API as exists today, and surprisingly minimal disruption to existing code.

Recap

Above is a rough view of Mitogen’s process model, revealing a desirable symmetry as it currently exists. In the master program and replicated children, the user’s code maintains full control of the main thread, with library communication requirements handled by a background thread using an identical implementation in every process.

Keeping the user in control of the main thread is important, as it possesses certain magical privileges. In Python it is the only thread from which signal handlers can be installed or executed, and on Linux some niche system interfaces require its participation.

When a method like remote_host.call(myfunc) is invoked, an outgoing message is constructed and enqueued with the Broker thread, and a callback handler is installed to cause any return value response message to be posted to another queue created especially to receive it. Meanwhile the thread that invoked Context.call(..) sleeps waiting for a message on the call’s dedicated reply queue.

Latches

Those queues aren’t simply Queue.Queue, but a custom reimplementation added early during Ansible extension development, as deficiencies in Python 2.x threading began to manifest. Python 2 permits the choice between up to 50 ms latency added to each Queue.get(), or for waits to execute with UNIX signals masked, thus preventing CTRL+C from interrupting the program. Given these options a reimplementation made plentiful sense.

The custom queue is called Latch, a name chosen simply because it was short and vaguely fitted. To say its existence is a great discomfort would be an understatement: reimplementing synchronization was never desired, even if just by leveraging OS facilities. True to tribal wisdom, the folly of Latch has been a vast time sink, costing many days hunting races and subtle misbehaviours, yet without it, good performance and usability is not possible on Python 2, and so it remains for now.

Due to this, when any thread blocks waiting for a result from a remote process, it always does so within Latch, a detail that will soon become important.

The Broker

Threading requirements are mostly due to Broker, a thread that has often changed role over time. Today its main function is to run an I/O multiplexer, like Twisted or asyncio. Except for some local file IO in master processes, broker thread code is asynchronous, regardless of whether it is communicating with a remote machine via an SSH subprocess or a local thread via a Latch.

When a user’s thread is blocked on a reply queue, that thread isn’t really blocked on a remote process – it is waiting for the broker thread to receive and decode any reply, then post it to the queue (or Latch) the thread is sleeping on.

Performance

Having a dedicated IO thread in a multi-threaded environment simplifies reasoning about communication, as events like unexpected disconnection always occur in a consistent location far from user code. But as is evident, it means every IO requires interaction of two threads in the local process, and when that communication is with a remote Mitogen process, a further two in the remote process.

It may come as no surprise that poor interaction with the OS scheduler often manifests, where load balancing pushes related communicating threads out across distinct cores, where their execution schedule bears no resemblance to the inherent lock-step communication pattern caused by the request-reply structure of RPCs, and between threads of the same process due to the Global Interpreter Lock. The range of undesirable effects defies simple description, it is sufficient to say that poor behaviour here can be disastrous.

To cope with this, the Ansible extension introduced CPU pinning. This feature locks related threads to one core, so that as a user thread enters a wait on the broker after sending it a message, the broker has much higher chance of being scheduled expediently, and for its use of shared resources (like the GIL) to be uncontended and exist in the cache of the CPU it runs on.

Runs of tests/bench/roundtrip.py with and without pinning.
Pinned? Round-trip delay
No

960 usec

Average 848 usec ± 111 usec

No

782 usec

No

803 usec

Yes

198 usec

Average 197 usec ± 1 usec

Yes

197 usec

Yes

197 usec

It is hard to overstate the value of pinning, as revealed by the 20% speedup visible in this stress test, but enabling it is a double-edged sword, as the scheduler loses the freedom to migrate processes to balance load, and no general pinning strategy is possible that does not approach the complexity of an entirely new scheduler. As a simple example, if two uncooperative processes (such as Ansible and, say, a database server) were to pin their busiest workers to the same CPU, both will suffer disastrous contention for resources that a scheduler could alleviate if it were permitted.

While performance loss due to scheduling could be considered a scheduler bug, it could be argued that expecting consistently low latency lock-step communication between arbitrary threads is unreasonable, and so it is desirable that threading rather than scheduling be considered at fault, especially as one and not the other is within our control.

The desire is not to remove threading entirely, but instead provide an option to disable it where it makes sense. For example in Ansible, it is possible to almost halve the running threads if worker processes were switched to a threadless implementation, since there is no benefit in the otherwise single-threaded WorkerProcess from having a distinct broker thread.

UNIX fork()

In its UNIX manifestation, fork() is a defective abstraction protected by religious symbolism and dogma, conceptualized at a time long predating the 1984 actualization of the problem it failed to solve. A full description of this exceeds any one paragraph, and an article in drafting since October already in excess of 8,000 words has not yet succeeded in fully capturing it.

For our purposes it is sufficient to know that, as when mixed with most UNIX facilities, mixing fork() with threads is extremely unsafe, but many UNIX programs presently rely on it, such as in Ansible’s forking of per-task worker processes. For that reason in the Ansible extension, Mitogen cannot be permanently active in the top-level process, but only after fork within a “connection multiplexer” subprocess, and within the per-task workers.

In upcoming work, there is a renewed desire for a broker to be active in the top-level process, but this is extremely difficult while remaining compatible with Ansible’s existing forking model. A threadless mode would be immediately helpful there.

Python 2.4

Another manifestation of fork() trouble comes in Python 2.4, where the youthful implementation makes no attempt to repair its threading state after fork, leading to incurable deadlocks across the board. For this reason when running on Python 2.4, the Ansible extension disables its internal use of fork for isolation of certain tasks, but it is not enough, as deadlocks while starting subprocesses are also possible.

A common idea would be to forget about Python 2.4 as it is too old, much as it is tempting to imagine HTTP 0.9 does not exist, but as in that case, Mitogen treats Python not just as a language runtime, but as an established network protocol that much be complied with in order to communicate with infrastructure that will continue to exist long into the future.

Implementation Approach

Recall it is not possible for a user thread to block without waiting on a Latch. With threadless mode, we can instead reinterpret the presence of a waiting Latch as the user’s indication some network IO is pending, and since the user cannot become unblocked until that IO is complete, and has given up forward execution in favour of waiting, Latch.get() becomes the only location where the IO loop must run, and only until the Latch that caused it to run has some result posted to it by the previous iteration.

@mitogen.main(threadless=True) def main(router):     host1 = router.ssh(hostname='a.b.c')     host2 = router.ssh(hostname='c.b.a')      call1 = host1.call_async(os.system, 'hostname')     call2 = host2.call_async(os.system, 'hostname')      print call1.get().unpickle()     print call2.get().unpickle() 

In the example, after the (presently blocking) connection procedure completes, neither call_async() wakes any broker thread, as none exists. Instead they enqueue messages for the broker to run, but the broker implementation does not start execution until call1.get(), where get() is internally synchronized using Latch.

The broker loop ceases after a result becomes available for the Latch that is executing it, only to be restarted again for call2.get(), where it again runs until its result is available. In this way asynchronous execution progresses opportunistically, and only when the calling thread indicated it cannot progress until a result is available.

Owing to the inconvenient existence of Latch, an initial prototype was functional with only a 30 line change. In this way, an ugly and undesirable custom synchronization primitive has accidentally become the centrepiece of an important new feature.

Size Benefit

The intention is that threadless mode will become the new default in a future version. As it has much lower synchronization requirements, it becomes possible to move large pieces of code out of the bootstrap, including any relating to implementing the UNIX self-pipe trick, as required by Latch, and to wake the broker thread from user threads.

Instead this code can be moved to a new mitogen.threads module, where it can progressively upgrade an existing threadless mitogen.core, much like mitogen.parent already progressively upgrades it with an industrial-strength Poller as required.

Any code that can be removed from the bootstrap has an immediate benefit on cold start performance with large numbers of targets, as the bottleneck during cold start is often a restriction on bandwidth.

Restrictions

Naturally this will place some restraints on execution. Transparent routing will no longer be quite so transparent, as it is not possible to execute a function call in a remote process that is also acting as a proxy to another process: proxying will not run while Dispatcher is busy executing the function call.

One simple solution is to start an additional child of the proxying process in which function calls will run, leaving its parent dedicated just to routing, i.e. exclusively dedicated to running what was previously the broker thread. It is expected this will require only a few lines of additional code to add support for in the Ansible extension.

For children of a threadless master, import statements will hang while the master is otherwise busy, but this is not much of a problem, since import statements usually happen once shortly after the first parent->child call, when the master will be waiting in a Latch.

For threadless children, no background thread exists to notice a parent has disconnected, and to ensure the process shuts down gracefully in case the main thread has hung. Some options are possible, including starting a subprocess for the task, or supporting SIGIO-based asynchronous IO, so the broker thread has can run from the signal handler and notice the parent is gone.

Another restriction is that when threadless mode is enabled, Mitogen primitives cannot be used from multiple threads. After some consideration, while possible to support, it does not seem worth the complexity, and would prevent the previously mentioned reduction of bootstrap code size.

Ongoing Work

Mitogen has quite an ugly concept of Services, added in a hurry during the initial Ansible extension development. Services represent a bundle of a callable method exposed to the network, a security policy determining who may call it, and an execution policy governing its concurrency requirements.

Despite heavy use, it has always been an ugly feature as it partially duplicates the normal parent->child function call mechanism. Looking at services from the perspective of threadless mode reveals some notion of a “threadless service”, and how such a threadless service looks even more similar to a function call than previously.

It is possible that as part of the threadless work, the unification of function calls and services may finally happen, although no design for it is certain yet.

Summary

There are doubtlessly many edge cases left to discover, but threadless mode looks very doable, and promises to make Mitogen suitable in even more scenarios than before.

Until next time!

Just tuning in?

Planet Python

Talk Python to Me: #199 Automate all the things with Python at Zapier

Do your applications call a lot of APIs? Maybe you have a bunch of microservices driving your app. You probably don’t have the crazy combinatorial explosion that Zapier does for connecting APIs! They have millions of users automating things with 1,000s of APIs. It’s pretty crazy. And they are doing it all with Python. Join me and Bryan Helmig, the CTO and co-founder of Zapier as we discuss how they pull this off with Python.
Planet Python

Real Python: Supercharge Your Classes With Python super()

While Python isn’t purely an object-oriented language, it’s flexible enough and powerful enough to allow you to build your applications using the object-oriented paradigm. One of the ways in which Python achieves this is by supporting inheritance, which it does with super().

In this tutorial, you’ll learn about the following:

  • The concept of inheritance in Python
  • Multiple inheritance in Python
  • How the super() function works
  • How the super() function in single inheritance works
  • How the super() function in multiple inheritance works

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you’ll need to take your Python skills to the next level.

An Overview of Python’s super() Function

If you have experience with object-oriented languages, you may already be familiar with the functionality of super().

If not, don’t fear! While the official documentation is fairly technical, at a high level super() gives you access to methods in a superclass from the subclass that inherits from it.

super() alone returns a temporary object of the superclass that then allows you to call that superclass’s methods.

Why would you want to do any of this? While the possibilities are limited by your imagination, a common use case is building classes that extend the functionality of previously built classes.

Calling the previously built methods with super() saves you from needing to rewrite those methods in your subclass, and allows you to swap out superclasses with minimal code changes.

super() in Single Inheritance

If you’re unfamiliar with object-oriented programming concepts, inheritance might be an unfamiliar term. Inheritance is a concept in object-oriented programming in which a class derives (or inherits) attributes and behaviors from another class without needing to implement them again.

For me at least, it’s easier to understand these concepts when looking at code, so let’s write classes describing some shapes:

class Rectangle:     def __init__(self, length, width):         self.length = length         self.width = width      def area(self):         return self.length * self.width      def perimeter(self):         return 2 * self.length + 2 * self.width  class Square:     def __init__(self, length):         self.length = length      def area(self):         return self.length * self.length      def perimeter(self):         return 4 * self.length 

Here, there are two similar classes: Rectangle and Square.

You can use them as below:

>>>

>>> square = Square(4) >>> square.area() 16 >>> rectangle = Rectangle(2,4) >>> rectangle.area() 8 

In this example, you have two shapes that are related to each other: a square is a special kind of rectangle. The code, however, doesn’t reflect that relationship and thus has code that is essentially repeated.

By using inheritance, you can reduce the amount of code you write while simultaneously reflecting the real-world relationship between rectangles and squares:

class Rectangle:     def __init__(self, length, width):         self.length = length         self.width = width      def area(self):         return self.length * self.width      def perimeter(self):         return 2 * self.length + 2 * self.width  # Here we declare that the Square class inherits from the Rectangle class class Square(Rectangle):     def __init__(self, length):         super().__init__(length, length) 

Here, you’ve used super() to call the __init__() of the Rectangle class, allowing you to use it in the Square class without repeating code. Below, the core functionality remains after making changes:

>>>

>>> square = Square(4) >>> square.area() 16 

In this example, Rectangle is the superclass, and Square is the subclass.

Because the Square and Rectangle .__init__() methods are so similar, you can simply call the superclass’s .__init__() method (Rectangle.__init__()) from that of Square by using super(). This sets the .length and .width attributes even though you just had to supply a single length parameter to the Square constructor.

When you run this, even though your Square class doesn’t explicitly implement it, the call to .area() will use the .area() method in the superclass and print 16. The Square class inherited .area() from the Rectangle class.

Note: To learn more about inheritance and object-oriented concepts in Python, be sure to check out Object-Oriented Programming (OOP) in Python 3.

What Can super() Do for You?

So what can super() do for you in single inheritance?

Like in other object-oriented languages, it allows you to call methods of the superclass in your subclass. The primary use case of this is to extend the functionality of the inherited method.

In the example below, you will create a class Cube that inherits from Square and extends the functionality of .area() (inherited from the Rectangle class through Square) to calculate the surface area and volume of a Cube instance:

class Square(Rectangle):     def __init__(self, length):         super().__init__(length, length)  class Cube(Square):     def surface_area(self):         face_area = super().area()         return face_area * 6      def volume(self):         face_area = super().area()         return face_area * self.length 

Now that you’ve built the classes, let’s look at the surface area and volume of a cube with a side length of 3:

>>>

>>> cube = Cube(3) >>> cube.surface_area() 54 >>> cube.volume() 27 

Caution: Note that in our example above, super() alone won’t make the method calls for you: you have to call the method on the proxy object itself.

Here you have implemented two methods for the Cube class: .surface_area() and .volume(). Both of these calculations rely on calculating the area of a single face, so rather than reimplementing the area calculation, you use super() to extend the area calculation.

Also notice that the Cube class definition does not have an .__init__(). Because Cube inherits from Square and .__init__() doesn’t really do anything differently for Cube than it already does for Square, you can skip defining it, and the .__init__() of the superclass (Square) will be called automatically.

super() returns a delegate object to a parent class, so you call the method you want directly on it: super().area().

Not only does this save us from having to rewrite the area calculations, but it also allows us to change the internal .area() logic in a single location. This is especially in handy when you have a number of subclasses inheriting from one superclass.

A super() Deep Dive

Before heading into multiple inheritance, let’s take a quick detour into the mechanics of super().

While the examples above (and below) call super() without any parameters, super() can also take two parameters: the first is the subclass, and the second parameter is an object that is an instance of that subclass.

First, let’s see two examples showing what manipulating the first variable can do, using the classes already shown:

class Rectangle:     def __init__(self, length, width):         self.length = length         self.width = width      def area(self):         return self.length * self.width      def perimeter(self):         return 2 * self.length + 2 * self.width  class Square(Rectangle):     def __init__(self, length):         super(Square, self).__init__(self, length, length) 

In Python 3, the super(Square, self) call is equivalent to the parameterless super() call. The first parameter refers to the subclass Square, while the second parameter refers to a Square object which, in this case, is self. You can call super() with other classes as well:

class Cube(Square):     def surface_area(self):         face_area = super(Square, self).area()         return face_area * 6      def volume(self):         face_area = super(Square, self).area()         return face_area * self.length 

In this example, you are setting Square as the subclass argument to super(), instead of Cube. This causes super() to start searching for a matching method (in this case, .area()) at one level above Square in the instance hierarchy, in this case Rectangle.

In this specific example, the behavior doesn’t change. But imagine that Square also implemented an .area() function that you wanted to make sure Cube did not use. Calling super() in this way allows you to do that.

Caution: While we are doing a lot of fiddling with the parameters to super() in order to explore how it works under the hood, I’d caution against doing this regularly.

The parameterless call to super() is recommended and sufficient for most use cases, and needing to change the search hierarchy regularly could be indicative of a larger design issue.

What about the second parameter? Remember, this is an object that is an instance of the class used as the first parameter. For an example, isinstance(Cube, Square) must return True.

By including an instantiated object, super() returns a bound method: a method that is bound to the object, which gives the method the object’s context such as any instance attributes. If this parameter is not included, the method returned is just a function, unassociated with an object’s context.

For more information about bound methods, unbound methods, and functions, read the Python documentation on its descriptor system.

Note: Technically, super() doesn’t return a method. It returns a proxy object. This is an object that delegates calls to the correct class methods without making an additional object in order to do so.

super() in Multiple Inheritance

Now that you’ve worked through an overview and some examples of super() and single inheritance, you will be introduced to an overview and some examples that will demonstrate how multiple inheritance works and how super() enables that functionality.

Multiple Inheritance Overview

There is another use case in which super() really shines, and this one isn’t as common as the single inheritance scenario. In addition to single inheritance, Python supports multiple inheritance, in which a subclass can inherit from multiple superclasses that don’t necessarily inherit from each other (also known as sibling classes).

I’m a very visual person, and I find diagrams are incredibly helpful to understand concepts like this. The image below shows a very simple multiple inheritance scenario, where one class inherits from two unrelated (sibling) superclasses:

A diagrammed example of multiple inheritanceA diagrammed example of multiple inheritance (Image: Kyle Stratis)

To better illustrate multiple inheritance in action, here is some code for you to try out, showing how you can build a right pyramid (a pyramid with a square base) out of a Triangle and a Square:

class Triangle:     def __init__(self, base, height):         self.base = base         self.height = height      def area(self):         return 0.5 * self.base * self.height  class RightPyramid(Triangle, Square):     def __init__(self, base, slant_height):         self.base = base         self.slant_height = slant_height      def area(self):         base_area = super().area()         perimeter = super().perimeter()         return 0.5 * perimeter * self.slant_height + base_area 

Note: The term slant height may be unfamiliar, especially if it’s been a while since you’ve taken a geometry class or worked on any pyramids.

The slant height is the height from the center of the base of an object (like a pyramid) up its face to the peak of that object. You can read more about slant heights at WolframMathWorld.

This example declares a Triangle class and a RightPyramid class that inherits from both Square and Triangle.

You’ll see another .area() method that uses super() just like in single inheritance, with the aim of it reaching the .perimeter() and .area() methods defined all the way up in the Rectangle class.

Note: You may notice that the code above isn’t using any inherited properties from the Triangle class yet. Later examples will fully take advantage of inheritance from both Triangle and Square.

The problem, though, is that both superclasses (Triangle and Square) define a .area(). Take a second and think about what might happen when you call .area() on RightPyramid, and then try calling it like below:

>>>

>> pyramid = RightPyramid(2, 4) >> pyramid.area() Traceback (most recent call last):   File "shapes.py", line 63, in <module>     print(pyramid.area())   File "shapes.py", line 47, in area     base_area = super().area()   File "shapes.py", line 38, in area     return 0.5 * self.base * self.height AttributeError: 'RightPyramid' object has no attribute 'height' 

Did you guess that Python will try to call Triangle.area()? This is because of something called the method resolution order.

Note: How did we notice that Triangle.area() was called and not, as we hoped, Square.area()? If you look at the last line of the traceback (before the AttributeError), you’ll see a reference to a specific line of code:

return 0.5 * self.base * self.height 

You may recognize this from geometry class as the formula for the area of a triangle. Otherwise, if you’re like me, you might have scrolled up to the Triangle and Rectangle class definitions and seen this same code in Triangle.area().

Method Resolution Order

The method resolution order (or MRO) tells Python how to search for inherited methods. This comes in handy when you’re using super() because the MRO tells you exactly where Python will look for a method you’re calling with super() and in what order.

Every class has an .__mro__ attribute that allows us to inspect the order, so let’s do that:

>>>

>>> RightPyramid.__mro__ (<class '__main__.RightPyramid'>, <class '__main__.Triangle'>,   <class '__main__.Square'>, <class '__main__.Rectangle'>,   <class 'object'>) 

This tells us that methods will be searched first in Rightpyramid, then in Triangle, then in Square, then Rectangle, and then, if nothing is found, in object, from which all classes originate.

The problem here is that the interpreter is searching for .area() in Triangle before Square and Rectangle, and upon finding .area() in Triangle, Python calls it instead of the one you want. Because Triangle.area() expects there to be a .height and a .base attribute, Python throws an AttributeError.

Luckily, you have some control over how the MRO is constructed. Just by changing the signature of the RightPyramid class, you can search in the order you want, and the methods will resolve correctly:

class RightPyramid(Square, Triangle):     def __init__(self, base, slant_height):         self.base = base         self.slant_height = slant_height         super().__init__(self.base)      def area(self):         base_area = super().area()         perimeter = super().perimeter()         return 0.5 * perimeter * self.slant_height + base_area 

Notice that RightPyramid initializes partially with the .__init__() from the Square class. This allows .area() to use the .length on the object, as is designed.

Now, you can build a pyramid, inspect the MRO, and calculate the surface area:

>>>

>>> pyramid = RightPyramid(2, 4) >>> RightPyramid.__mro__ (<class '__main__.RightPyramid'>, <class '__main__.Square'>,  <class '__main__.Rectangle'>, <class '__main__.Triangle'>,  <class 'object'>) >>> pyramid.area() 20.0 

You see that the MRO is now what you’d expect, and you can inspect the area of the pyramid as well, thanks to .area() and .perimeter().

There’s still a problem here, though. For the sake of simplicity, I did a few things wrong in this example: the first, and arguably most importantly, was that I had two separate classes with the same method name and signature.

This causes issues with method resolution, because the first instance of .area() that is encountered in the MRO list will be called.

When you’re using super() with multiple inheritance, it’s imperative to design your classes to cooperate. Part of this is ensuring that your methods are unique so that they get resolved in the MRO, by making sure method signatures are unique—whether by using method names or method parameters.

In this case, to avoid a complete overhaul of your code, you can rename the Triangle class’s .area() method to .tri_area(). This way, the area methods can continue using class properties rather than taking external parameters:

class Triangle:     def __init__(self, base, height):         self.base = base         self.height = height         super().__init__()      def tri_area(self):         return 0.5 * self.base * self.height 

Let’s also go ahead and use this in the RightPyramid class:

class RightPyramid(Square, Triangle):     def __init__(self, base, slant_height):         self.base = base         self.slant_height = slant_height         super().__init__(self.base)      def area(self):         base_area = super().area()         perimeter = super().perimeter()         return 0.5 * perimeter * self.slant_height + base_area      def area_2(self):         base_area = super().area()         triangle_area = super().tri_area()         return triangle_area * 4 + base_area 

The next issue here is that the code doesn’t have a delegated Triangle object like it does for a Square object, so calling .area_2() will give us an AttributeError since .base and .height don’t have any values.

You need to do two things to fix this:

  1. All methods that are called with super() need to have a call to their superclass’s version of that method. This means that you will need to add super().__init__() to the .__init__() methods of Triangle and Rectangle.

  2. Redesign all the .__init__() calls to take a keyword dictionary. See the complete code below.

class Rectangle:     def __init__(self, length, width, **kwargs):         self.length = length         self.width = width         super().__init__(**kwargs)      def area(self):         return self.length * self.width      def perimeter(self):         return 2 * self.length + 2 * self.width  # Here we declare that the Square class inherits from  # the Rectangle class class Square(Rectangle):     def __init__(self, length, **kwargs):         super().__init__(length=length, width=length, **kwargs)  class Cube(Square):     def surface_area(self):         face_area = super().area()         return face_area * 6      def volume(self):         face_area = super().area()         return face_area ** 3  class Triangle:     def __init__(self, base, height, **kwargs):         self.base = base         self.height = height         super().__init__(**kwargs)      def tri_area(self):         return 0.5 * self.base * self.height  class RightPyramid(Square, Triangle):     def __init__(self, base, slant_height, **kwargs):         self.base = base         self.slant_height = slant_height         kwargs["height"] = slant_height         kwargs["length"] = base         super().__init__(base=base, **kwargs)      def area(self):         base_area = super().area()         perimeter = super().perimeter()         return 0.5 * perimeter * self.slant_height + base_area      def area_2(self):         base_area = super().area()         triangle_area = super().tri_area()         return triangle_area * 4 + base_area 

There are a number of important differences in this code:

  • kwargs is modified in some places (such as RightPyramid.__init__()): This will allow users of these objects to instantiate them only with the arguments that make sense for that particular object.

  • Setting up named arguments before **kwargs: You can see this in RightPyramid.__init__(). This has the neat effect of popping that key right out of the **kwargs dictionary, so that by the time that it ends up at the end of the MRO in the object class, **kwargs is empty.

Note: Following the state of kwargs can be tricky here, so here’s a table of .__init__() calls in order, showing the class that owns that call, and the contents of kwargs during that call:

Class Named Arguments kwargs
RightPyramid base, slant_height
Square length base, height
Rectangle length, width base, height
Triangle base, height

Now, when you use these updated classes, you have this:

>>>

>>> pyramid = RightPyramid(base=2, slant_height=4) >>> pyramid.area() 20.0 >>> pyramid.area_2() 20.0 

It works! You’ve used super() to successfully navigate a complicated class hierarchy while using both inheritance and composition to create new classes with minimal reimplementation.

Multiple Inheritance Alternatives

As you can see, multiple inheritance can be useful but also lead to very complicated situations and code that is hard to read. It’s also rare to have objects that neatly inherit everything from more than multiple other objects.

If you see yourself beginning to use multiple inheritance and a complicated class hierarchy, it’s worth asking yourself if you can achieve code that is cleaner and easier to understand by using composition instead of inheritance.

With composition, you can add very specific functionality to your classes from a specialized, simple class called a mixin.

Since this article is focused on inheritance, I won’t go into too much detail on composition and how to wield it in Python, but here’s a short example using VolumeMixin to give specific functionality to our 3D objects—in this case, a volume calculation:

class Rectangle:     def __init__(self, length, width):         self.length = length         self.width = width      def area(self):         return self.length * self.width  class Square(Rectangle):     def __init__(self, length):         super().__init__(length, length)  class VolumeMixin:     def volume(self):         return self.area() * self.height  class Cube(VolumeMixin, Square):     def __init__(self, length):         super().__init__(length)         self.height = length      def area(self):         return super().area() * 6 

In this example, the code was reworked to include a mixin called VolumeMixin. The mixin is then used by Cube and gives Cube the ability to calculate its volume, which is shown below:

>>>

>>> cube = Cube(2) >>> cube.area() 24 >>> cube.volume() 48 

This mixin can be used the same way in any class that has an area defined for it and for which the formula area * height returns the correct volume.

A super() Recap

In this tutorial, you learned how to supercharge your classes with super(). Your journey started with a review of single inheritance and then showed how to call superclass methods easily with super().

You then learned how multiple inheritance works in Python, and techniques to combine super() with multiple inheritance. You also learned about how Python resolves method calls using the method resolution order (MRO), as well as how to inspect and modify the MRO to ensure appropriate methods are called at appropriate times.

For more information about object-oriented programming in Python and using super(), check out these resources:


[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Planet Python