8.3 Exceptions#

Estimated time for this notebook: 15 minutes

When we learned about testing, we saw that Python complains when things go wrong by raising an “Exception” naming a type of error:

1 / 0
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
Cell In[1], line 1
----> 1 1 / 0

ZeroDivisionError: division by zero

Exceptions are objects, forming a class hierarchy. We just raised an instance of the ZeroDivisionError class, making the program crash. If we want more information about where this class fits in the hierarchy, we can use Python’s inspect module to get a chain of classes, from ZeroDivisionError up to object:

import inspect

inspect.getmro(ZeroDivisionError)
(ZeroDivisionError, ArithmeticError, Exception, BaseException, object)

So we can see that a zero division error is a particular kind of Arithmetic Error.

x = 1

for y in x:
    print(y)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[3], line 3
      1 x = 1
----> 3 for y in x:
      4     print(y)

TypeError: 'int' object is not iterable
inspect.getmro(TypeError)
(TypeError, Exception, BaseException, object)

Create your own Exception#

When we were looking at testing, we saw that it is important for code to crash with a meaningful exception type when something is wrong. We raise an Exception with raise. Often, we can look for an appropriate exception from the standard set to raise.

However, we may want to define our own exceptions. Doing this is as simple as inheriting from Exception (or one of its subclasses):

class MyCustomErrorType(ArithmeticError):
    pass


raise MyCustomErrorType("Problem")
---------------------------------------------------------------------------
MyCustomErrorType                         Traceback (most recent call last)
Cell In[5], line 5
      1 class MyCustomErrorType(ArithmeticError):
      2     pass
----> 5 raise MyCustomErrorType("Problem")

MyCustomErrorType: Problem

You can add custom data to your exception:

class MyCustomErrorType(Exception):
    def __init__(self, category=None):
        self.category = category

    def __str__(self):
        return f"Error, category {self.category}"


raise MyCustomErrorType(404)
---------------------------------------------------------------------------
MyCustomErrorType                         Traceback (most recent call last)
Cell In[6], line 9
      5     def __str__(self):
      6         return f"Error, category {self.category}"
----> 9 raise MyCustomErrorType(404)

MyCustomErrorType: Error, category 404

The real power of exceptions comes, however, not in letting them crash the program, but in letting your program handle them. We say that an exception has been “thrown” and then “caught”.

import yaml

try:
    config = yaml.safe_load(open("datasource.yaml"))
    user = config["userid"]
    password = config["password"]

except FileNotFoundError:
    print("No password file found, using anonymous user.")
    user = "anonymous"
    password = None


print(user)
No password file found, using anonymous user.
anonymous

Note that we specify only the error we expect to happen and want to handle. Sometimes you see code that catches everything:

try:
    config = yaml.safe_lod(open("datasource.yaml"))
    user = config["userid"]
    password = config["password"]
except:
    user = "anonymous"
    password = None

print(user)
anonymous

This can be dangerous and can make it hard to find errors! There was a mistyped function name there (’safe_lod’), but we did not notice the error, as the generic except caught it. Therefore, we should be specific and catch only the type of error we want.

Managing multiple exceptions#

Let’s create two credential files to read

with open("datasource2.yaml", "w") as outfile:
    outfile.write("userid: eidle\n")
    outfile.write("password: secret\n")

with open("datasource3.yaml", "w") as outfile:
    outfile.write("user: eidle\n")
    outfile.write("password: secret\n")

And create a function that reads credentials files and returns the username and password to use.

def read_credentials(source):
    try:
        datasource = open(source)
        config = yaml.safe_load(datasource)
        user = config["userid"]
        password = config["password"]
        datasource.close()
    except FileNotFoundError:
        print("Password file missing")
        user = "anonymous"
        password = None
    except KeyError:
        print("Expected keys not found in file")
        user = "anonymous"
        password = None
    return user, password
print(read_credentials("datasource2.yaml"))
('eidle', 'secret')
print(read_credentials("datasource.yaml"))
Password file missing
('anonymous', None)
print(read_credentials("datasource3.yaml"))
Expected keys not found in file
('anonymous', None)

This last code has a flaw: the file was successfully opened, the missing key was noticed, but not explicitly closed. It’s normally OK, as Python will close the file as soon as it notices there are no longer any references to datasource in memory, after the function exits. But this is not good practice, you should keep a file handle for as short a time as possible.

def read_credentials(source):
    try:
        datasource = open(source)
        config = yaml.safe_load(datasource)

        try:
            print("File loaded, trying to extract credentials")
            user = config["userid"]
            password = config["password"]
        except KeyError:
            print("Expected keys not found in file")
            user = "anonymous"
            password = None
        finally:
            # Runs irrespective of whether keys found
            print("Closing file")
            datasource.close()

    except FileNotFoundError:
        print("Password file missing")
        user = "anonymous"
        password = None

    return user, password

The finally clause is executed whether or not an exception occurs.

The last optional clause of a try statement, an else clause is called only if an exception is NOT raised. It can be a better place than the try clause to put code other than that which you expect to raise the error, and which you do not want to be executed if the error is raised. It is executed in the same circumstances as code put in the end of the try block, the only difference is that errors raised during the else clause are not caught.

def read_credentials(source):
    try:
        datasource = open(source)

    except FileNotFoundError:
        print("Password file missing")
        user = "anonymous"
        password = None

    else:
        # Runs only if opening the file was successful
        config = yaml.safe_load(datasource)
        try:
            print("File loaded, trying to extract credentials")
            user = config["userid"]
            password = config["password"]
        except KeyError:
            print("Expected keys not found in file")
            user = "anonymous"
            password = None
        finally:
            # Runs irrespective of whether keys found
            print("Closing file")
            datasource.close()

    return user, password

Don’t worry if else seems useless to you; most languages’ implementations of try/except don’t support such a clause. An alternative way of avoiding leaving the file open in the original implementation (and without using else or finally) is to use a context manager:

def read_credentials(source):
    try:
        with open(source) as datasource:  # closes the file when done
            config = yaml.safe_load(datasource)
        user = config["userid"]
        password = config["password"]
    except FileNotFoundError:
        print("Password file missing")
        user = "anonymous"
        password = None
    except KeyError:
        print("Expected keys not found in file")
        user = "anonymous"
        password = None
    return user, password

Catching Exceptions Elsewhere#

Exceptions do not have to be caught close to the part of the program calling them. They can be caught anywhere “above” the calling point in the call stack: control can jump arbitrarily far in the program: up to the except clause of the “highest” containing try statement.

def f4(x):
    if x == 0:
        return
    if x == 1:
        raise ArithmeticError()
    if x == 2:
        raise SyntaxError()
    if x == 3:
        raise TypeError()
def f3(x):
    try:
        print("F3Before")
        f4(x)
        print("F3After")
    except ArithmeticError:
        print("F3Except (💣)")
def f2(x):
    try:
        print("F2Before")
        f3(x)
        print("F2After")
    except SyntaxError:
        print("F2Except (💣)")
def f1(x):
    try:
        print("F1Before")
        f2(x)
        print("F1After")
    except TypeError:
        print("F1Except (💣)")
f1(0)
F1Before
F2Before
F3Before
F3After
F2After
F1After
f1(1)
F1Before
F2Before
F3Before
F3Except (💣)
F2After
F1After
f1(2)
F1Before
F2Before
F3Before
F2Except (💣)
F1After
f1(3)
F1Before
F2Before
F3Before
F1Except (💣)

Design with Exceptions#

Now we know how exceptions work, we need to think about the design implications… How best to use them.

Traditional software design theory will tell you that they should only be used to describe and recover from exceptional conditions: things going wrong. Normal program flow shouldn’t use them.

Python’s designers take a different view: use of exceptions in normal flow is considered OK. For example, all iterators raise a StopIteration exception to indicate the iteration is complete.

A commonly recommended Python design pattern is to use exceptions to determine whether an object implements a protocol (concept/interface), rather than testing on type.

For example, we might want a function which can be supplied either a data series or a path to a location on disk where data can be found. We can examine the type of the supplied content:

import yaml


def analysis(source):
    if type(source) == dict:
        name = source["modelname"]
    else:
        content = open(source)
        source = yaml.safe_load(content)
        name = source["modelname"]
    print(name)
analysis({"modelname": "Super"})
Super
with open("example.yaml", "w") as outfile:
    outfile.write("modelname: brilliant\n")
analysis("example.yaml")
brilliant

However, we can also use the try-it-and-handle-exceptions approach to this.

def analysis(source):
    try:
        name = source["modelname"]
    except TypeError:
        content = open(source)
        source = yaml.safe_load(content)
        name = source["modelname"]
    print(name)


analysis("example.yaml")
brilliant

This approach is more extensible, and behaves properly if we give it some other data-source which responds like a dictionary or string.

def analysis(source):
    try:
        name = source["modelname"]
    except TypeError:
        # Source was not a dictionary-like object
        # Maybe it is a file path
        try:
            content = open(source)
            source = yaml.safe_load(content)
            name = source["modelname"]
        except IOError:
            # Maybe it was already raw YAML content
            source = yaml.safe_load(source)
            name = source["modelname"]
    print(name)


analysis("modelname: Amazing")
Amazing

Re-Raising Exceptions#

Sometimes we want to catch an error, partially handle it, perhaps add some extra data to the exception, and then re-raise to be caught again further up the call stack.

The keyword “raise” with no argument in an except: clause will cause the caught error to be re-thrown. Doing this is the only circumstance where it is safe to do except: without catching a specific type of error.

try:
    # Something
    pass
except:
    # Do this code here if anything goes wrong
    raise

If you want to be more explicit about where the error came from, you can use the raise from syntax, which will create a chain of exceptions:

def lower_function():
    raise ValueError("Error in lower function!")


def higher_function():
    try:
        lower_function()
    except ValueError as e:
        raise RuntimeError("Error in higher function!") from e


higher_function()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[32], line 7, in higher_function()
      6 try:
----> 7     lower_function()
      8 except ValueError as e:

Cell In[32], line 2, in lower_function()
      1 def lower_function():
----> 2     raise ValueError("Error in lower function!")

ValueError: Error in lower function!

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
Cell In[32], line 12
      8     except ValueError as e:
      9         raise RuntimeError("Error in higher function!") from e
---> 12 higher_function()

Cell In[32], line 9, in higher_function()
      7     lower_function()
      8 except ValueError as e:
----> 9     raise RuntimeError("Error in higher function!") from e

RuntimeError: Error in higher function!

It can be useful to catch and re-throw an error as you go up the chain, doing any clean-up needed for each layer of a program.

The error will finally be caught and not re-thrown only at a higher program layer that knows how to recover. This is known as the “throw low catch high” principle.