02_intermediate_python

Open on binder:

Call out to Corey Schafer

Lots of the code here is taken from Corey Schafer and his excellent python tutorials. When you are more the video-learning type of person, definitely check out his youtube channel:

https://www.youtube.com/user/schafer5

What you will learn

In this tutorial you will learn:

• How to add functionality to your iterations and make theam more readable.

• How to make your classes awesome.

  • How to inherit from parent classes.

  • How to make your classes work with builtin operators (operator overloading).

  • How to write decorators in classes and on functions.

Advanced iteration

There are multiple objects than can be iterated. str, list, and tuple are the most common. We will later look at how to make our own object iterable. But first let’s go back to the iteration over a list:

iterable = [0, 1, 5, 'hello', 1.2, (1, 'world')]
for i in iterable:
    print(i)

0
1
5
hello
1.2
(1, 'world')

Any and all

You can use the builtin function any() or all() to check if any object or all objects are True. Because all objects have a boolean value assigned to them, you can do stuff like this:

check = any(['', 0, 0.0, None, False, [], {}, ()])
print(check)

check = any(['', 0, 0.0, None, False, [], ~False])
print(check)

False
True

all([True, True, True, True])

True

Bitwise not. It will, for example invert a binary number and return its compliment.

b = 0b100111111
print(bin(b))
print(bin(~b))

0b100111111
-0b101000000

Combining lists

iterable2 = [i**2 for i in range(len(iterable))]
for i in iterable + iterable2:
    print(i)

0
1
5
hello
1.2
(1, 'world')
0
1
4
9
16
25

Enumerate

With enumerate() we get the ability to get the current iteration count without doing something like this:

i = 0
for j in something:
    important_variable = something_else[i]
    i += 1

enumerate() returns an iterator that gives tuples when it is iterated over.

for tup in enumerate(iterable):
    print(tup)

(0, 0)
(1, 1)
(2, 5)
(3, 'hello')
(4, 1.2)
(5, (1, 'world'))

The returned tuple can be directly unpacked.

for i, val in enumerate(iterable):
    print(f"Value at {i} is: {val}")

Value at 0 is: 0
Value at 1 is: 1
Value at 2 is: 5
Value at 3 is: hello
Value at 4 is: 1.2
Value at 5 is: (1, 'world')

Zip

With zip(), two iterables can be iterated through at the same time or they can be iterated one after the other. The number of iterations by zip correspond to the number of elements of the shortest itearable. zip() also gives a tuple which can directly unpacked.

Zip through two lists

for i, j in zip(iterable, iterable2):
    print(i, j)

0 0
1 1
5 4
hello 9
1.2 16
(1, 'world') 25

Zip with enumerate

zip and enumerate can be used simultaneously, but you have to watch, how you unpack everything.

for i, (val1, val2) in enumerate(zip(iterable, iterable2)):
    print(f"Value1at {i} is: {val1}. Value2 is {val2}")

Value1at 0 is: 0. Value2 is 0
Value1at 1 is: 1. Value2 is 1
Value1at 2 is: 5. Value2 is 4
Value1at 3 is: hello. Value2 is 9
Value1at 4 is: 1.2. Value2 is 16
Value1at 5 is: (1, 'world'). Value2 is 25

Constructing a dict from two lists using zip

We can construct a dict using zip like this:

weird_dict = dict(zip(iterable2, iterable))

BUT, because normally the keys in a dict are str we have to make the values in iterable2 to str. Let me introduce you to the built-in function map(). With map() we can map any function to an iterable. map() returns a map object. This map object is iterable itself. But if you want to create a list from it, you have to call the list() built-in function, which creates lists from iterables.

Showcase of the ``list()`` built-in function

list('Hello world!')

['H', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd', '!']

Map the bui-in function ``str()`` to the iterable2 list

tmp = map(str, iterable2)
print(tmp)
tmp = list(tmp)
print(tmp)

<map object at 0x7f5b68e0b520>
['0', '1', '4', '9', '16', '25']

Finally. Built the dict

weird_dict = dict(zip(tmp, iterable))
print(weird_dict)
print(weird_dict['9'])

{'0': 0, '1': 1, '4': 5, '9': 'hello', '16': 1.2, '25': (1, 'world')}
hello

Map

The builtin function map() applies a function to every element of an iterable. Python3.x returns a map object (This is done for performance reasons. The execution of the function square() is halted until the result is truly needed. This saves memory and computation time, especially, if you only want to use the first n items). To create a list from this, pass it to the list() builtin.

def square(num):
    return num **2

map(square, [1, 2, 3, 4])

<map at 0x7f5b68d9ddf0>

list(map(square, [1, 2, 3, 4]))

[1, 4, 9, 16]

%%timeit
map(square, [1, 2, 3, 4])

209 ns ± 1.45 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

%%timeit
list(map(square, [1, 2, 3, 4]))

1.41 µs ± 2.26 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

Filter

Instead of mapping a function to be applied to every element. You can give a function, that either returns True or False. Elements which would cause the function to return False are dropped from the iterable.

strings = ['Spam', 'Foo', 'Cat', 'Bar', 'Noodles', 'Parrot', 'List', 'Hello', 'Window', 'World']

def filter_programmer_humor(inp):
    humor = ['spam', 'foo', 'bar', 'parrot', 'hello', 'world']
    if inp.lower() in humor:
        return False
    else:
        return True

list(filter(filter_programmer_humor, strings))

['Cat', 'Noodles', 'List', 'Window']

Reduce

Reduce is a function from the functools builtin library. It provides powerful rolling computation for something that you would use a for-loop otherwise.

That way, you can calculate the geometric mean of a set of numbers.

from functools import reduce

def multiplication(x, y):
    return x * y

values = [1, 2, 3, 4, 5, 6]

print(reduce(multiplication, values) ** (1/len(values)))

2.993795165523909

Even more advanced iteration, combination, permutation (package: itertools)

Check out the built-in package itertools to get more iterators, like combinations and permutations.

Advanced functions *args and **kwargs

If you’ve looked at some python code, you’ve certainly come across these two expressions. Let’s demystify them.

Taken from: https://realpython.com/python-kwargs-and-args/

The *args argument in a function definition allows any number of arguments to be passed. Additional to a defined number of arguments declared beforehand.

def summation(prt, *args):
    """Prints the first argument and sums the remaining args."""
    print(f"Argument prt got value: {prt}")
    out = 0
    for i in args:
        out += i
    return out

summation('I think. At some point you should take a break', 2, 3, 5, 6, 7)

Argument prt got value: I think. At some point you should take a break

23

We can supply any arguments from iterables by unpacking them in the function call.

summation(*range(120))

Argument prt got value: 0

7140

The unpacking elements can be calles whatever, but it is advised to use *args and **kwargs

employee_numbers = {'Nancy': 20, 'Keith': 5, 'Sandra': 10, 'Ben': 5}
department_members = ['Nancy', 'Keith', 'Richard']

def add_department(members, **kwargs):
    """Adds numbers belonging to department members."""
    sum_ = 0
    for member in kwargs:
        print(f"{member} is in department.")
        if member in department_members:
            sum_ = sum_ + kwargs[member]
    return sum_

add_department(department_members, **employee_numbers, Richard=7)

Nancy is in department.
Keith is in department.
Sandra is in department.
Ben is in department.
Richard is in department.

32

There’s a lot going on in the last cell. Let’s summarize it:

• The first line defines a dictionary with multiple employees and some numbers belonging to them.

• The second line defines a list of strings. The members of a specific department.

• After the docstring in the function a variable called sum_ is set to zero. The trailing underscore is added, so the built-in function sum() is not overwritten with the value zero.

• The members in the kwargs argument are iterated over.

• An if-check is made to see whether the member is in the list members.

• If that’s the case the value of the key is added to sum_

When the function was called an impromptu employee was added as a keyworded argument. He also gets passed into the for member in kwargs for-loop.

Overwriting default values

The **kwargs can be used to define standard values for the function, which convienently can be overwritten by an input **kwargs.

department_members = ['Nancy', 'Keith', 'Richard', 'Ben']

def calculate_new_salary(member, **kwargs):
    """Calculate new salaries for employees."""
    defaults = {'payment': 50000, 'increase': 1.05}
    for key in kwargs:
        if key not in defaults and not isinstance(kwargs[key], int):
            raise ValueError(f"Can't parse arguments {key}")

    # replace values in default with kwargs
    data = {**defaults, **kwargs}
    pay = data['payment']
    increase = data['increase']

    # get the remaining **kwargs not in defaults
    boni = sum([value for key, value in data.items() if key not in defaults])

    new_salary = int(pay * increase + boni)
    print(f"{member} gets {new_salary} €.")

for member in department_members:
    calculate_new_salary(member)

calculate_new_salary('Greg, the manager', payment=10000, increase=1.15)

calculate_new_salary('John, the CEO', payment=150000, increase=1.05, bonus1=5000, bonus2=8500)

# add_department(department_members, **employee_numbers, Richard=7)

Nancy gets 52500 €.
Keith gets 52500 €.
Richard gets 52500 €.
Ben gets 52500 €.
Greg, the manager gets 11500 €.
John, the CEO gets 171000 €.

Advanced Classes OOP

The strength of python lies in the easy creation of new classes and their object-oriented nature.

Why OOP?

• Modularity for easier troubleshooting

• Reuse of code through inheritance

• Flexibility through polymorphism

• Effective problem solving

Instance variables

There are class variables and instance variables. Consider this example with a class defining an employee. Let’s start at the most basic level:

class Employee:
    pass

emp1 = Employee()
emp2 = Employee()

emp1.first = 'Diana'
emp1.last = 'Prince'
emp1.email = 'diana.price@company.com'
emp1.pay = 5000

emp2.first = 'Clark'
emp2.last = 'Kent'
emp2.email = 'clark.kene@company.com'
emp2.pay = 6000

print(emp1, emp2)
print(emp1.first, emp1.last, emp1.email, emp1.pay)
print(emp2.first, emp2.last, emp2.email, emp2.pay)

<__main__.Employee object at 0x7f5b68d9d160> <__main__.Employee object at 0x7f5b68d9d220>
Diana Prince diana.price@company.com 5000
Clark Kent clark.kene@company.com 6000

What we have done here is: - Created a class called Employee. - Created two instances of this class (emp1 and emp2), also called instantiating. - Created some variables of these instances (variables of instances are also called attributes).

Instead of setting the instance variables, tediously one after the other for all instances, we can define a placeholder for the current instance in our classes. By convention the instance is called self and this instance needs to be the first argument to all methods in our class. The next step is thus:

class Employee:

    def __init__(self, first, last):
        self.first = first
        self.last = last

emp1 = Employee('Diana', 'Price')
emp2 = Employee('Clark', 'Kent')

print(emp1.first, emp1.last)
print(emp2.first, emp2.last)

Diana Price
Clark Kent

Notice how we saved some lines. Also notice how we assigned the input arguments first and last to the instance variables self.first and self.last. Wihtout these selfs, the variables would not be accessible from outside of the method inside the class. We will now add the email and the pay to __init__(self):

class Employee:

    def __init__(self, first, last, pay):
        self.first = first
        self.last = last
        self.pay = pay
        self.email = f'{first.lower()}.{last.lower()}@company.com'

emp1 = Employee('Diana', 'Price', 5000)
emp2 = Employee('Clark', 'Kent', 6000)

print(emp1.email)
print(emp2.email)

diana.price@company.com
clark.kent@company.com

Now let’s say we want to perform some actions with our classes. This is done by methods. These methods are like functions inside the scope of a class. And they always take the instance (convention self) as first argument. Let’s say we want to print the full name of an employee:

class Employee:

    def __init__(self, first, last, pay):
        self.first = first
        self.last = last
        self.pay = pay
        self.email = f'{first.lower()}.{last.lower()}@company.com'

    def fullname(self):
        return f'{self.first} {self.last}'

emp1 = Employee('Diana', 'Price', 5000)
emp2 = Employee('Clark', 'Kent', 6000)

print(emp1.fullname())
print(emp2.fullname())

Diana Price
Clark Kent

This might look confusing, because we don’t pass any arguments into the fullname() method of the emp1 instance. but as previously said, Class methods automatically take the instance as first arguemnt. What is really happening is this:

emp1 = Employee('Diana', 'Price', 5000)
fullname = Employee.fullname(emp1)
print(fullname)

Diana Price

Here, we provided the instance (emp1) to the method fullname() of the Employee class and because the Employee class is not an instance, we need to provide the instance ourselves.

Class variables

Class variables are shared between all instances of the class. Let’s say our company raises the payout every year and always by a factor. That would be a good idea to put into a class variable. Let’s first try to hardcode the raise with instance variables and see why class variables are better later.

class Employee:

    def __init__(self, first, last, pay):
        self.first = first
        self.last = last
        self.pay = pay
        self.email = f'{first.lower()}.{last.lower()}@company.com'

    def fullname(self):
        return f'{self.first} {self.last}'

    def apply_raise(self):
        self.pay = int(self.pay * 1.04)

emp1 = Employee('Diana', 'Price', 5000)
emp2 = Employee('Clark', 'Kent', 6000)

print(emp1.pay)
emp1.apply_raise()
print(emp1.pay)

5000
5200

Let’s say we want to alter the raise factor by doing something like:

emp1.raise_amount
Employee.raise_amount

This 4% raise amount (times 1.04) can easily set as an class variable. Inside the apply_raise() mehtod we could either use the instance (self):

def apply_raise(self):
    self.pay = int(self.pay * self.raise_amount)

or use the class Employee itself:

def apply_raise(self):
    self.pay = int(self.pay * Employee.raise_amount)

Which is better?

Depends.

The first one would allow us to increase the pay_amount of one Employee:

class Employee:

    raise_amount = 1.04

    def __init__(self, first, last, pay):
        self.first = first
        self.last = last
        self.pay = pay
        self.email = f'{first.lower()}.{last.lower()}@company.com'

    def fullname(self):
        return f'{self.first} {self.last}'

    def apply_raise(self):
        self.pay = int(self.pay * self.raise_amount)

emp1 = Employee('Diana', 'Price', 5000)
emp2 = Employee('Clark', 'Kent', 6000)

print(Employee.raise_amount)
print(emp1.raise_amount)
print(emp2.raise_amount)

emp1.raise_amount = 1.06

print(Employee.raise_amount)
print(emp1.raise_amount)
print(emp2.raise_amount)

1.04
1.04
1.04
1.04
1.06
1.04

The second one would allow us to increase the raise_amount of all employees at the same time.

class Employee:

    raise_amount = 1.04

    def __init__(self, first, last, pay):
        self.first = first
        self.last = last
        self.pay = pay
        self.email = f'{first.lower()}.{last.lower()}@company.com'

    def fullname(self):
        return f'{self.first} {self.last}'

    def apply_raise(self):
        self.pay = int(self.pay * Employee.raise_amount)

emp1 = Employee('Diana', 'Price', 5000)
emp2 = Employee('Clark', 'Kent', 6000)

print(Employee.raise_amount)
print(emp1.raise_amount)
print(emp2.raise_amount)

Employee.raise_amount = 1.06

print(Employee.raise_amount)
print(emp1.raise_amount)
print(emp2.raise_amount)

1.04
1.04
1.04
1.06
1.06
1.06

As a second example for class variables, where it makes no sense to use the instance self as a namespace, but the class would be a counter that counts how many instances there are of one class.

class Employee:

    raise_amount = 1.04
    num_of_emps = 0

    def __init__(self, first, last, pay):
        self.first = first
        self.last = last
        self.pay = pay
        self.email = f'{first.lower()}.{last.lower()}@company.com'
        Employee.num_of_employees += 1

    def fullname(self):
        return f'{self.first} {self.last}'

    def apply_raise(self):
        self.pay = int(self.pay * self.raise_amount)

emp1 = Employee('Diana', 'Price', 5000)
emp2 = Employee('Clark', 'Kent', 6000)
emp3 = Employee('Wade', 'Wilson', -20)

print(Employee.num_of_emps)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[31], line 19
     16     def apply_raise(self):
     17         self.pay = int(self.pay * self.raise_amount)
---> 19 emp1 = Employee('Diana', 'Price', 5000)
     20 emp2 = Employee('Clark', 'Kent', 6000)
     21 emp3 = Employee('Wade', 'Wilson', -20)

Cell In[31], line 11, in Employee.__init__(self, first, last, pay)
      9 self.pay = pay
     10 self.email = f'{first.lower()}.{last.lower()}@company.com'
---> 11 Employee.num_of_employees += 1

AttributeError: type object 'Employee' has no attribute 'num_of_employees'

Inheritance

Inheritance allows us to inherit attribnutes and method of parent classes into child classes and overwrite attributes and methods as we like. Let’s say we want to create different type of employees: developers and managers. Both have names and emails. So we can reuse the code from the Employee class. With parentheses after the class statement we can inherit from other classes.

class Employee:

    raise_amount = 1.04

    def __init__(self, first, last, pay):
        self.first = first
        self.last = last
        self.pay = pay
        self.email = f'{first.lower()}.{last.lower()}@company.com'

    def fullname(self):
        return f'{self.first} {self.last}'

    def apply_raise(self):
        self.pay = int(self.pay * self.raise_amount)


class Developer(Employee):
    pass


dev1 = Developer('Diana', 'Price', 5000)
dev2 = Developer('Clark', 'Kent', 6000)

print(dev1.email)
print(dev2.email)

diana.price@company.com
clark.kent@company.com

We want to customize our child classes. Let’s first see, what happens, if we raise the pay of developers.

print(dev1.pay)
dev1.apply_raise()
print(dev1.pay)

5000
5200

Let’s say we want out Developers to have a higher raise_amount:

class Developer(Employee):
    raise_amount = 1.1
    pass

dev1 = Developer('Diana', 'Price', 5000)
print(dev1.pay)
dev1.apply_raise()
print(dev1.pay)

5000
5500

You can see, that the pay increased by 10%. Our Employees are untouched by this change. Let’s make a few more changes. Let’s say we have additional information for Developers, like their current programming language. Instead of writing a completely new function (the email attribute and the fullname() methoud should stay the same), we can give the Developer class its own __init__() method.

class Developer(Employee):

    raise_amount = 1.1

    def __init__(self, first, last, pay, prog_lang):
        super().__init__(first, last, pay)
        self.prog_lang = prog_lang

With the super().__init__() method, we make the Employee class handle the pay and email and everything that happens in the __init__() method of the Employee class. The Developer class then does its own stuff with the prog_lang.

dev1 = Developer('Diana', 'Price', 5000, 'python')
dev2 = Developer('Clark', 'Kent', 6000, 'java')

print(dev1.email)
print(dev1.prog_lang)

diana.price@company.com
python

Let’s look at another example using class inheritance

class Polygon:
    def __init__(self, no_of_sides):
        self.n = no_of_sides
        self.sides = [0 for i in range(self.n)]
        print(f"{self.__class__.__name__} init.")

    def inputSides(self, *args):
        # take an undefined number of args
        if args:
            self.sides = [i for i in args]
        else:
            self.sides = [float(input(f"Enter {self.__class__.__name__}'s side {i+1} length: ")) for i in range(self.n)]

    def dispSides(self):
        for i in range(self.n):
            print(f"Side {i+1} of {self.__class__.__name__} is {self.sides[i]} length units long.")

    def _findArea(self):
        raise Exception("Area of arbitrary Polygon not defined.")

    def printArea(self):
        self._findArea()
        print(f"The area of the {self.__class__.__name__.lower()} is {self.area:.2f} area units.")

quadrilateral = Polygon(4)
quadrilateral.inputSides(7, 8, 9, 19)
quadrilateral.dispSides()

Polygon init.
Side 1 of Polygon is 7 length units long.
Side 2 of Polygon is 8 length units long.
Side 3 of Polygon is 9 length units long.
Side 4 of Polygon is 19 length units long.

The Triangle class inherits from the Polygon class.

In the Triangle class’ __init__() method, the __init__() method of the parent class needs to be called, this can be done with either:

class Child(Base):
    def __init__(self):
        Base.__init__(self)
    ...
    ...

However, if we do it this way, the lookup for methods might get scrambled, when we have multiple inheritances (So a Base class, a Child_A class, that inherits from Base and a Child_B class, that inherits from Child_A). With the builtin super() method that Method-Resolution-Order (MRO) is like expected: First look for the method in Child_B, then Child_A, then Base.

class Child(Base):
    def __init__(self):
        super().__init__()
    ...
    ...

class Triangle(Polygon):
    def __init__(self):
        super().__init__(3) # pass 3 to the parent class' __init__

    def _findArea(self):
        a, b, c = self.sides
        s = (a + b + c) / 2
        self.area = (s*(s-a)*(s-b)*(s-c)) ** 0.5

tri = Triangle()
tri.inputSides(2, 3, 4)
tri.printArea()

Triangle init.
The area of the triangle is 2.90 area units.

Let’s create a rectangle and include checks for the side lengths.

class Rectangle(Polygon):
    def __init__(self):
        super().__init__(4) # pass 4 to the parent class' __init__

    def inputSides(self, *args):
        # check the length of the sides
        if args:
            assert args[0] == args[2], "Side a and c of a rectangle need to be of equal length."
            assert args[1] == args[3], "Side b and d of a rectangle need to be of equal length."
        super().inputSides(*args) # call the parent class' inputSides() method.

    def _findArea(self):
        self.area = self.sides[0] * self.sides[1]

rect = Rectangle()
rect.inputSides(2, 3, 2, 3)
rect.printArea()

Rectangle init.
The area of the rectangle is 6.00 area units.

There are some checks builtin to class inheritance for your convenience.

print(isinstance(tri, Triangle)) # standard isinstance builtin function

print(issubclass(Rectangle, Polygon)) # there is also a issubclass() builtin

print(issubclass(bool, int)) # interestingly, bool is a subclass of int. Whomst'd've !

True
True
True

Double under methods

These double under methods also called magic methods allow our classes to work with some builtin functions and also use operator overloading. Here is a short overview over these so-called dunder methods:

Creation __new__

The __new__ method can be used to either execute the class’ __init__ method or not. Sometimes you don’t want to instantiate an instance from a class, because of something. You can control that with the __new__ method. It takes the class (convention cls) as first argument and needs all other arguments, that __init__ also takes.

class Shape:
    def __new__(cls, sides, *args, **kwargs):
        if sides == 3:
            return Triangle(*args, **kwargs)
        else:
            return Square(*args, **kwargs)


class Triangle:
    def __init__(self, base, height):
        self.base = base
        self.height = height

    def area(self):
        return (self.base * self.height) / 2


class Square:
    def __init__(self, length):
        self.length = length

    def area(self):
        return self.length*self.length


a = Shape(sides=3, base=2, height=12)
b = Shape(sides=4, length=2)

print(str(a.__class__))
print(a.area())

print(str(b.__class__))
print(b.area())

<class '__main__.Triangle'>
12.0
<class '__main__.Square'>
4

Representation __str__ and __repr__

The __str__() and __repr__() Methods are what’s called when the class is printed and returned, respectively.

class MyClass:
    def __init__(self, integer=1):
        self.int = integer
        pass

    def __repr__(self):
        return "<%s at memory address 0x%02x>" % (self.__str__(), id(self))

    def __str__(self):
        return f'Instance of MyClass. Contains the integer {self.int}.'

myclass = MyClass()
print(myclass)

Instance of MyClass. Contains the integer 1.

myclass

<Instance of MyClass. Contains the integer 1. at memory address 0x7f5b68c5ec70>

Types __bool__

You can assign your custom class a boolean value:

class Items:
    def __init__(self, items=None):
        if items is None:
            self.items = []
        else:
            self.items = items

    def __bool__(self):
        return len(self.items) != 0

items1 = Items()
items2 = Items(['pen', 'book', 'binder'])

print(bool(items1))
print(bool(items2))

False
True

Indexing __len__ , __getitem__ and __reversed__

With these dunder methods you can make your items iterable for item in iterable, indexable class[index] and also let the buildin function reversed() act on your classes.

class MyClass:
    def __init__(self, some_numbers):
        # sort the numbers into negative and postive numbers
        self.positive = [i for i in some_numbers if i >= 0]
        self.negative = [i for i in some_numbers if i < 0]

    def __len__(self):
        """Gives the length of positive and negative numbers."""
        return len(self.positive) + len(self.negative)

    def __getitem__(self, key):
        """-1 gives negative numbers, +1 gives positive.
        Otherwise this __getitem__ is pretty stupid.

        """
        if key == -1:
            return self.negative
        elif key == 1:
            return self.positive
        else:
            raise IndexError(f'MyClass index can either be -1 or 1. You provided {key}')

    def __reversed__(self):
        """Return a new instance with negative and positives swapped.
        Also a pretty stupid idea.

        """
        new_instance = MyClass(self.positive + self.negative)
        new_instance.negative, new_instance.positive = self.positive, self.negative
        return new_instance

my_numbers = [-5, -7, 0, 1, 5, -10, 15, -4, -2, -5, 5, 10]

myclass = MyClass(my_numbers)

print(len(myclass), len(my_numbers))
print(myclass[-1])
print(myclass[1])

notmyclass = reversed(myclass)
print(notmyclass[-1])

12 12
[-5, -7, -10, -4, -2, -5]
[0, 1, 5, 15, 5, 10]
[0, 1, 5, 15, 5, 10]

Comparing __eq__ and __lt__ and so on.

With these dunder methods you can use the comparison operators with your custom classes.

class Animal:

    animal_weights = {'blue whale': 136000, 'giraffe': 800, 'polar bear': 475,
                      'red deer': 200, 'tiger': 120, 'cheetah': 54, 'chimpanzee': 45,
                      'coyote': 14, 'nutria': 8}

    def __init__(self, animal):
        self.animal = animal
        self.weight = Animal.animal_weights[animal]

    def __eq__(self, other):
        return self.weight == other.weight

    def __lt__(self, other):
        return self.weight < other.weight

    def __gt__(self, other):
        return self.weight > other.weight

    def __ge__(self, other):
        return self.weight >= other.weight

    def __le__(self, other):
        return self.weight <= other.weight

    def __ne__(self, other):
        return self.weight != other.weight

animal1 = Animal('blue whale')
animal2 = Animal('polar bear')
animal3 = Animal('nutria')
animal4 = Animal('polar bear')

print(animal1 == animal2)
print(animal1 > animal2)
print(animal1 < animal2)
print(animal2 == animal4)
print(animal2 >= animal4)
print(animal2 <= animal4)

False
True
False
True
True
True

Merging __add__

With this dunder method, you can choose what happens, if the + operator is called with your class. Here, you are free to choose what happens. You don’t need to add similar instances:

class NewAnimal(Animal):
    def __add__(self, other):
        self.weight += other
        return self

animal = NewAnimal('polar bear')
print(animal.weight)
animal += 50
print(animal.weight)

475
525

But most often, it is advised, that __add__ takes the same class as self. But the returned class doesn’t need to be the same.

class Herd:
    def __init__(self, animals):
        self.animals = animals
        self.weight = sum([animal.weight for animal in animals])

    def __add__(self, other):
        if isinstance(other, Herd):
            animals = self.animals + other.animals
        if isinstance(other, HerdAnimal):
            animals = self.animals + [other]
        return Herd(animals)

    def __str__(self):
        return f"Herd with {len(self.animals)} animals"

class HerdAnimal(Animal):
    def __add__(self, other):
        return Herd([self, other])

animal1 = HerdAnimal('blue whale')
animal2 = HerdAnimal('polar bear')
animal3 = HerdAnimal('nutria')

herd = animal1 + animal2 + animal3
print(herd)

Herd with 3 animals

Callable objects __call__

You can call instances. Calls are always done by parentheses () in python. Just like when you call functions.

class Division:
    def __init__(self, dividend):
        self.dividend = dividend

    def __call__(self, x):
        return self.dividend / x

div = Division(10)
print(div(5))

2.0

Context Manager __enter__ and __exit__

To open files in python you can do something like this:

f = open("test.txt")
data = f.read()
f.close()
line1 = data[0]
...
...

However, you might already have come across this syntax:

with open("test.txt") as f:
    data = f.read()
# more code
line1 = data[0]
...
...

This is the new syntax, which replaces the old open() and close() builtin function and file-object method. The with expression starts what is called a context manager. Context managers are used when you want to free up system resources. Opened file objects and opened databases for example don’t need to stay in memory once the needed information has been extracted. Context managers also come in handy, because on linux systems the number of concurrently opened files is limited to 1024. To create a context manager for your class you need to add some magic, dunder methods.

Take a close look at how the __enter__() method returns the self instance of the class. This is the instance of the class, that will be used inside the context manager.

The arguments in the __exit__() method are used to deal with Exceptions. Take note, how instance variables are deleted to free up space.

class MyContextManager:
    def __init__(self):
        print('Init called')

    def __enter__(self):
        print("Enter method called. Populating scope...")
        self.int = 1
        self.str = 'Tim the Enchanter!'
        self.large_variable = [i*2 for i in range(1000)]
        return self

    def __exit__(self, exc_type, exc_value, exc_traceback):
        print('Exit method called.')
        del self.int
        del self.str
        del self.large_variable

    def divide_by_zero_err(self):
        return self.int / 0

with MyContextManager() as manager:
    print("Inside Context")
    print(manager.large_variable[20])

Init called
Enter method called. Populating scope...
Inside Context
40
Exit method called.

The instance variables str, int, and large_variable have been deleted, once the context is closed.

manager.str

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[55], line 1
----> 1 manager.str

AttributeError: 'MyContextManager' object has no attribute 'str'

What about the exc_type arguments? Those are to make sure the __exit__() method is called, although some Exception is encountered in the context manager. First the method with the arguments

Iteration __iter__ and __next__

These methods allow you to iterate over something in your custom class.

from string import ascii_letters, ascii_uppercase

class Alphabet:
    def __init__(self, alphabet=ascii_letters):
        self.alphabet = alphabet
        self._max = 24

    def __iter__(self):
        self._index = 0
        return self

    def __next__(self):
        if self._index >= self._max:
            raise StopIteration
        else:
            self._index += 1
            return self._index - 1, self.alphabet[self._index - 1]

alphabet1 = Alphabet()
alphabet2 = Alphabet(ascii_uppercase)

for number, letter in alphabet1:
    print(number, letter)

for number, letter in alphabet2:
    print(number, letter)

0 a
1 b
2 c
3 d
4 e
5 f
6 g
7 h
8 i
9 j
10 k
11 l
12 m
13 n
14 o
15 p
16 q
17 r
18 s
19 t
20 u
21 v
22 w
23 x
0 A
1 B
2 C
3 D
4 E
5 F
6 G
7 H
8 I
9 J
10 K
11 L
12 M
13 N
14 O
15 P
16 Q
17 R
18 S
19 T
20 U
21 V
22 W
23 X

Built-in decorators

Before going on some lengthy tangent about what a decorator is and how to write one on your own, let’s just one of the three built-in decorators:

• @property

• @setter

• @classmethod

• @staticmethod

The property decorator

The @property decorator is often used and adds a nice way of dynamically defining instance variables (also called attributes). Let’s reuse our Employee function and say don’t want to set the self.email attribute inside the __init__() method. What if the name of a employee changes? The email would still be the same. So let’s create a method called email()

class Employee:

    def __init__(self, first, last):
        self.first = first
        self.last = last

    def email(self):
        return f'{self.first.lower()}.{self.last.lower()}@company.com'

    def fullname(self):
        return f'{self.first} {self.last}'

emp_1 = Employee('Jane', 'Doe')
print(emp_1.email())

jane.doe@company.com

This is good and all, but now we need to call the email() method every time, we need the email. Some of the above code only uses the self.email attribute. Changing the class this way might break some of our old code. So let’s use the @propert decorator:

class Employee:

    def __init__(self, first, last):
        self.first = first
        self.last = last

    @property
    def email(self):
        return f'{self.first.lower()}.{self.last.lower()}@company.com'

    def fullname(self):
        return f'{self.first} {self.last}'

emp_1 = Employee('Jane', 'Doe')
print(emp_1.email)

jane.doe@company.com

And now self.email behaves just like an attribute. That’s the purpose of the @property decorator. The same could be done for the fullname() method. But let’s not stop there. With a @setter decorator we could achieve a piece of code that returns fullname as an attribute (without parentheses):

>>> print(emp_1.fullname)
Jane Doe

but also allows us to change the instance’s first and last arttributes by doing:

>>> emp_1.fullname = 'Jane Smith'
>>> print(emp_1.email)
jane.smith@company.com

The setter decorator

class Employee:

    def __init__(self, first, last):
        self.first = first
        self.last = last

    @property
    def email(self):
        return f'{self.first.lower()}.{self.last.lower()}@company.com'

    @property
    def fullname(self):
        return f'{self.first} {self.last}'

    @fullname.setter
    def fullname(self, new_fullname):
        self.first, self.last = new_fullname.split()

emp_1 = Employee('Jane', 'Doe')
print(emp_1.email)
emp_1.fullname = 'Jane Smith'
print(emp_1.email)

jane.doe@company.com
jane.smith@company.com

The deleter decorator

The code inside the deleter decorator is executed once the variable is accessed.

class Employee:

    def __init__(self, first, last):
        self.first = first
        self.last = last

    @property
    def email(self):
        return f'{self.first.lower()}.{self.last.lower()}@company.com'

    @property
    def fullname(self):
        return f'{self.first} {self.last}'

    @fullname.setter
    def fullname(self, new_fullname):
        self.first, self.last = new_fullname.split()

    @fullname.deleter
    def fullname(self):
        print("Deleting Name!")
        self.first = None
        self.last = None

emp_1 = Employee('Jane', 'Doe')
print(emp_1.email)
del emp_1.fullname
print(emp_1.first)

jane.doe@company.com
Deleting Name!
None

The classmethod decorator

Regular methods (also the __init__() method) take the instance of the class as first argument. By convention, we call the instance of the class inside the class itself self, but other variables are possible. There are, however some special decorators (more on decorators later) that you can use to circumvent this.

Class methods take the class as first argument (per convention cls) and return a new instance of the class, with something altered. For example, the raise_amount:

class Employee:

    raise_amount = 1.04
    num_of_emps = 0

    def __init__(self, first, last, pay):
        self.first = first
        self.last = last
        self.pay = pay
        self.email = f"{self.first.lower()}.{self.last.lower()}@company.com"

        # change the class variable of the Employee class everytime __init__ is called
        Employee.num_of_emps += 1

    def fullname(self):
        return f"{self.first} {self.last}"

    def apply_raise(self):
        self.pay = int(self.pay * self.raise_amount)

    @classmethod
    def set_raise_amount(cls, amount):
        cls.raise_amount = amount


emp1 = Employee('Clark', 'Kent', 50000)
emp2 = Employee('Bruce', 'Wayne', 60000)
print(emp1.pay, emp2.pay)
emp1.apply_raise(); emp2.apply_raise()
print(emp1.pay, emp2.pay, '\n')

# Apply changes class-wide
Employee.set_raise_amount(1.1)
emp1 = Employee('Clark', 'Kent', 50000)
emp2 = Employee('Bruce', 'Wayne', 60000)
print(emp1.pay, emp2.pay)
emp1.apply_raise(); emp2.apply_raise()
print(emp1.pay, emp2.pay)

50000 60000
52000 62400

50000 60000
55000 66000

Classes with different constructors

Let’s say, we want to create new employees from a string of the type: firstname-lastname-pay. We use the split() method of the string object to split it based on the hyphens.

We use a class method for this alternate constructor. Per convention, these methods start with from

str1 = 'Clark-Kent-50000'
str2 = 'Bruce-Wayne-60000'

first, last, pay = str1.split('-')
print(first, last, pay)

Clark Kent 50000

class Employee:

    raise_amount = 1.04
    num_of_emps = 0

    def __init__(self, first, last, pay):
        self.first = first
        self.last = last
        self.pay = pay
        self.email = f"{self.first.lower()}.{self.last.lower()}@company.com"

        # change the class variable of the Employee class everytime __init__ is called
        Employee.num_of_emps += 1

    def fullname(self):
        return f"{self.first} {self.last}"

    def apply_raise(self):
        self.pay = int(self.pay * self.raise_amount)

    @classmethod
    def set_raise_amount(cls, amount):
        cls.raise_amount = amount

    @classmethod
    def from_string(cls, employee_string):
        first, last, pay = employee_string.split('-')
        return cls(first, last, pay)

emp1 = Employee('Clark', 'Kent', 50000)
emp2 = Employee.from_string('Bruce-Wayne-60000')

print(emp1.email, emp2.email)

clark.kent@company.com bruce.wayne@company.com

Static method decorator

If there’s a method inside a class, that would also make sense outside the scope of the class, you can make it a static method. This static method can now be called directly from the class without having to create an instance. Note, how there is no self or cls in the is_workday() static method.

A good giveaway, whether a method should be a staticmethod is, when you don’t access the class, or the instance.

class Employee:

    raise_amount = 1.04
    num_of_emps = 0

    def __init__(self, first, last, pay):
        self.first = first
        self.last = last
        self.pay = pay
        self.email = f"{self.first.lower()}.{self.last.lower()}@company.com"

        # change the class variable of the Employee class everytime __init__ is called
        Employee.num_of_emps += 1

    def fullname(self):
        return f"{self.first} {self.last}"

    def apply_raise(self):
        self.pay = int(self.pay * self.raise_amount)

    @classmethod
    def set_raise_amount(cls, amount):
        cls.raise_amount = amount

    @classmethod
    def from_string(cls, employee_string):
        first, last, pay = employee_string.split('-')
        return cls(first, last, pay)

    @staticmethod
    def is_workday(day):
        if day.weekday() == 5 or day.weekday() == 6:
            return False
        return True

import datetime
my_date = datetime.date(2016, 7, 10)
print(Employee.is_workday(my_date))

emp1 = Employee.from_string('Clark-Kent-50000')
my_date = datetime.date(2016, 7, 11)
print(emp1.is_workday(my_date))

False
True

Custom decorators

Now that we have learned about the built-in decorators let’s write our own. For that we need to write a closure. A closure is a function that returns a function. The general style is as follows:

def outer(*args, **kwargs):
    def inner(*args, **kwargs):
        return args
    return inner

def arithmetic(operation='addition'):
    def addition(x, y):
        return x + y
    def subtraction(x, y):
        return x - y
    def multiplication(x, y):
        return x * y
    def division(x, y):
        return x / y
    return locals()[operation]

arithmetic('addition')(1, 2)
arithmetic('subtraction')(1, 2)

-1

But not only that, we need our outer function to accept a function as argument. Like so:

def my_decorator(func):
    def wrapper(*args, **kwargs):
        print("Do something before function call")
        func(*args, **kwargs)
        print("Do something after the function call")
    return wrapper

Let’s write a decorator that sorts the **kwargs alphabetically:

def sorted_kwargs(func):
    def wrapper(*args, **kwargs):
        kwargs = {key: value for key, value in sorted(kwargs.items())}
        return func(*args, **kwargs)
    return wrapper

def increase_price(increase=1.5, **kwargs):
    return {key: value * increase for key, value in kwargs.items()}

items = {'apple': 1, 'milk': 1.5, 'banana': 0.5, 'potatoes': 2.5}

# first without decorator
print(increase_price(**items))

# now with decorator so let's decorate our function
increase_price = sorted_kwargs(increase_price)
print(increase_price(**items))

{'apple': 1.5, 'milk': 2.25, 'banana': 0.75, 'potatoes': 3.75}
{'apple': 1.5, 'banana': 0.75, 'milk': 2.25, 'potatoes': 3.75}

Instead of defining our increase_price() function twice (once for creating the function and once for adding the decorator), we could add the decorator as a syntactic sugar with the known @decorator function:

@sorted_kwargs
def increase_price(increase=1.5, **kwargs):
    return {key: value * increase for key, value in kwargs.items()}

print(increase_price(**items))

{'apple': 1.5, 'banana': 0.75, 'milk': 2.25, 'potatoes': 3.75}

Useful use-cases

Do something more often.

def do_three_times(func):
    def wrapper(*args, **kwargs):
        func()
        func()
        func()
    return wrapper

@do_three_times
def print_hello_world():
    print("Hello world")

print_hello_world()

Hello world
Hello world
Hello world

Debug a function py printing *args and **kwargs:

def debug(func):
    def wrapper(*args, **kwargs):
        args_repr = [repr(a) for a in args]                      # 1
        kwargs_repr = [f"{k}={v!r}" for k, v in kwargs.items()]  # 2
        signature = ", ".join(args_repr + kwargs_repr)           # 3
        print(f"Calling {func.__name__}({signature})")
        value = func(*args, **kwargs)
        print(f"{func.__name__!r} returned {value!r}")           # 4
        return value
    return wrapper

@debug
def make_greeting(name, age=None):
    if age is None:
        return f"Howdy {name}!"
    else:
        return f"Whoa {name}! {age} already, you are growing up!"

make_greeting("Benjamin")

Calling make_greeting('Benjamin')
'make_greeting' returned 'Howdy Benjamin!'

'Howdy Benjamin!'

make_greeting("Richard", age=112)

Calling make_greeting('Richard', age=112)
'make_greeting' returned 'Whoa Richard! 112 already, you are growing up!'

'Whoa Richard! 112 already, you are growing up!'

Preserve identity with functools.wraps

We have a small problem, when using decorators this way:

print(make_greeting)
print(make_greeting.__name__)

<function debug.<locals>.wrapper at 0x7f5b6883a820>
wrapper

Our function now has the __name__ attribute wrapper. Ugh. That’s not good and can lead to some problems later down the line. That’s why it is recommended to use @functools.wraps(func) in decorator definitions:

import functools

def do_three_times(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        func()
        func()
        func()
    return wrapper

@do_three_times
def print_hello_world():
    print("Hello world")

print(print_hello_world)
print(print_hello_world.__name__)

<function print_hello_world at 0x7f5b6883af70>
print_hello_world

All good.

Decorator classes

We could use a decorator class to count how often a function that has this decorator has been called. For that we need the previously introduced __call__ method of python classes.

import functools

class CountCalls:
    def __init__(self, func):
        functools.update_wrapper(self, func)
        self.func = func
        self.num_calls = 0

    def __call__(self, *args, **kwargs):
        self.num_calls += 1
        print(f"Call {self.num_calls} of {self.func.__name__!r}")
        return self.func(*args, **kwargs)

@CountCalls
def say_whee():
    print("Whee!")

say_whee()

Call 1 of 'say_whee'
Whee!

say_whee()

Call 2 of 'say_whee'
Whee!

say_whee()

Call 3 of 'say_whee'
Whee!

say_whee()

Call 4 of 'say_whee'
Whee!

Further keywords

import keyword
print(keyword.kwlist)

['False', 'None', 'True', 'and', 'as', 'assert', 'async', 'await', 'break', 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal', 'not', 'or', 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield']

Recap: Builtins

Don’t confuse these Keywords with the builtin-functions, that you can access via:

(The filter function drops the Error, Warnings and Exception, we’ll look into later)

def drop_err_warn_exc(string):
    if not any(i in string for i in ['Error', 'Warning', 'Exception']):
        return True

non_error_builtins = list(filter(drop_err_warn_exc, [builtin for builtin in dir(__builtins__)]))
print(non_error_builtins)

['Ellipsis', 'False', 'GeneratorExit', 'KeyboardInterrupt', 'None', 'NotImplemented', 'StopAsyncIteration', 'StopIteration', 'SystemExit', 'True', '__IPYTHON__', '__build_class__', '__debug__', '__doc__', '__import__', '__loader__', '__name__', '__package__', '__spec__', 'abs', 'all', 'any', 'ascii', 'bin', 'bool', 'breakpoint', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'display', 'divmod', 'enumerate', 'eval', 'exec', 'execfile', 'filter', 'float', 'format', 'frozenset', 'get_ipython', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'map', 'max', 'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'range', 'repr', 'reversed', 'round', 'runfile', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars', 'zip']

Every python function returns None by default, which boolean value is False. If the function has been completely executed and nothing has been returned yet, a None will be returned.

Accidentally overwritten Builtins

You may find yourself in a situation, where you have accidentally overwritten one of the builtin functions. To restore them, you can simply use the del keyword to delete the variable, thus freeing up the namespace to be occupied again by the builtin.

print(sum)
sum = 'Run away!'
print(sum)
del sum
print(sum)

<built-in function sum>
Run away!
<built-in function sum>

Lambda expressions

Lambda expressions are used to generate fast ‘throwaway’ functions. The general syntax is as follows:

func = lambda var: expression_with(var)
square = lambda x: x **2

For multiple arguments use this syntax:

multiplication = lambda x, y: x * y

If you want your lambda function to return True or False for a filter() function call, for example, you can use syntax similar to this:

contains_vowels = lambda x: True if not any(s in x for s in ['a', 'e', 'i', 'o', 'u']) else False

func = lambda var: var**2
func(2)

4

contains_vowels = lambda x: True if not any(s in x for s in ['a', 'e', 'i', 'o', 'u']) else False
contains_vowels('Hello')

False

filter_func = lambda string: True if not any(i in string for i in ['Error', 'Warning', 'Exception']) else False

non_error_builtins = list(filter(filter_func ,[builtin for builtin in dir(__builtins__)]))

print(non_error_builtins)

['Ellipsis', 'False', 'GeneratorExit', 'KeyboardInterrupt', 'None', 'NotImplemented', 'StopAsyncIteration', 'StopIteration', 'SystemExit', 'True', '__IPYTHON__', '__build_class__', '__debug__', '__doc__', '__import__', '__loader__', '__name__', '__package__', '__spec__', 'abs', 'all', 'any', 'ascii', 'bin', 'bool', 'breakpoint', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'display', 'divmod', 'enumerate', 'eval', 'exec', 'execfile', 'filter', 'float', 'format', 'frozenset', 'get_ipython', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'map', 'max', 'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'range', 'repr', 'reversed', 'round', 'runfile', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars', 'zip']

Global and Nonlocal -> Scope and Namespace

Python’s namespace is the mapping from object to name. So if we call:

a = 2

we map an object (remember: everything in python is an object) called int, or integer to the variable a. This creates a name, that is accessible in the global namespace. The namespaces are sorted hierarchically and you can remember them with the abbreviation LEGB for Local - Enclosing - Global - Built-in.

Scope is a term which describes which namespace a call looks into. A function call can for example look into the Built-in namespace (you can call list() from within a function, without declaring it. Top-level calls (like code in jupyter cells) can’t look into a function’s namespace. Their scope is limited to the Global and Built-in namespace.

Let’s get into some examples… We declare a global variable called x and assign it a string object.

x = 'global x'

def test():
    print(x)

test()

global x

The function test() has access to the global variable x.

x = 'global x'

def test():
    y = 5
    print(x, y)

test()
print(x, y)

global x 5

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[86], line 8
      5     print(x, y)
      7 test()
----> 8 print(x, y)

NameError: name 'y' is not defined

The global scope (including the global and builtin namespace) can’t access teh variable y, unless…

x = 'global x'

def test():
    global y
    y = 5
    print(x, y)

test()
print(x, y)

global x 5
global x 5

We declare y to be a global variable.

Arguments to functions are also part of the function’s scope and won’t be accessible to the outside.

x = 'global x'

def test(z):
    x = 'local x' # overwrite global
    print(x, z)

test('local z')

local x local z

What is the enclosing scope?

The enclosing scope is used for nested functions. Here is a simple example of nested functions. Note the call inner() of the inner function.

# global namespace
x = 'global x'

def outer():
    # local namespace
    x = 'outer x'

    def inner():
        # enclosed namespace
        x = 'inner x'
        print(x)

    inner()
    print(x)

outer()

inner x
outer x

Similar to top-level functions, the inner function has access to the outer functions namespace. We can look at this by commenting out the x = 'inner x' statement.

x = 'global x'

def outer():
    x = 'outer x'

    def inner():
        # x = 'inner x'
        print(x)

    inner()
    print(x)

outer()

outer x
outer x

And here’s where the nonlocal keyword comes into play. The nonlocal keyword overwrites the outer functions variable but leaves the global variable unchanged.

x = 'global x'

def outer():
    x = 'outer x'
    print(x)
    def inner():
        nonlocal x
        x = 'inner x'
        print(x)

    inner()
    print(x)

outer()
print(x)

outer x
inner x
inner x
global x

The keyword global however, does not affect the local scope of the function outer()

x = 'global x'

def outer():
    x = 'outer x'
    print(x)
    def inner():
        global x
        x = 'inner x'
        print(x)

    inner()
    print(x)

outer()
print(x)

outer x
inner x
outer x
inner x

Generators and yield

Generators are your way of making python a little bit faster. We’ve already seen how the builtins range() and map() don’t actually return a list. They also use generators to keep the calculation of something on hold until it is really needed.

>>> out = map(func, [1, 2, 3, 4])
>>> print(out)
<map object at 0x7f2d5c4385c0>

We can create our own generators using the yield keyword. Consider this example. We have a function that returns a list of square numbers. And the generator function, making use of the yield keyword. Let’s say we want to find out, which number’s square is greater than 100.

def square_iterator(nums):
    result = []
    for i in nums:
        result.append(i * i)
    return result

def square_generator(nums):
    for i in nums:
        yield i * i

greater_than = 50
my_nums = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
my_squares = square_iterator(my_nums)
index = list(map(lambda x: True if x > greater_than else False, my_squares)).index(True)
print(f"The first integer whose square is greater than {greater_than} is '{my_nums[index]}'. The square is '{my_squares[index]}'")

The first integer whose square is greater than 50 is '8'. The square is '64'

The function call square_iterator(my_nums) makes python calculate the square of all the nums in the list my_nums. However, it would be great if we can just stop the calculation when we reach 100, without writing a new function (that is writing a function specifically for the purpose of finding this number. A function which returns the squares of a list has a much broader application).

For this we need to actually make the generator calculate stuff, because right now, it only returns a generator object.

generator = square_generator(my_nums)
print(generator)

<generator object square_generator at 0x7f5b689f3430>

The generator is ready to take the next result. For this, we’ll use the next() builtin function.

next(generator)

1

next(generator)

4

next(generator)

9

What happens, when we reach the end of the my_nums list?

generator = square_generator(my_nums)
while True:
    print(next(generator))

1
4
9
16
25
36
49
64
81
100
121
144
169
196
225

---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
Cell In[99], line 3
      1 generator = square_generator(my_nums)
      2 while True:
----> 3     print(next(generator))

StopIteration:

We get a StopIteration Exception (More on Errors later). The generator has been exhausted.

This is a very important differentiation between iterators (lists, tuples, dicts) and generators. Generators can become exhausted. Once all calculations have been done with the generator no values can be accessed any further.

However, generators can still be used as iterables in for loops. The StopIteration Exception automatically terminates the for loop. In this next example the result of the square_generator() function call is not assigned to a variable. This way we don’t have to worry about exhaustion.

for i in square_generator(my_nums):
    print(i)

1
4
9
16
25
36
49
64
81
100
121
144
169
196
225

With this, we can accelerate the search above. This function is much faster than the one above.

By defining the variable my_nums with a generator, rather than a list [1, 2, 3, 4, 5, 6, ...] we can save time by not typing the list explicitly. And we can pass as great a number, as we like into the greater_than variable.

def int_generator():
    i = 0
    while True:
        yield i
        i += 1

my_nums = int_generator()
generator = square_generator(my_nums)
i = 0
greater_than = 50000
while True:
    value = next(generator)
    if value > greater_than:
        break
    i += 1

print(f"The first integer whose square is greater than {greater_than} is '{i}'. The square is '{value}'")

The first integer whose square is greater than 50000 is '224'. The square is '50176'

Exceptions

The time has finally come to talk about Exceptions, Errors, Warnings and Exception Handling.

The words Exception and Error are often used interchangeably. However, it is pretty clear, that Exception is the base class. All other Exceptions are Errors, that inherit from Exception.

issubclass(NotImplementedError, Exception)

True

So Exception is the base class. How can we use it. We can use the builtin class Exception to stop the Execution of some code.

def integer_addition(*args):
    out = 0
    for i in args:
        if not isinstance(i, int):
            raise Exception(f"All provided types need to be <int>, you provided <{i.__class__.__name__}>.")
        out += i
    return out

integer_addition(1, 2, 3, 4, 5, ['this is a list'])

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[103], line 9
      6         out += i
      7     return out
----> 9 integer_addition(1, 2, 3, 4, 5, ['this is a list'])

Cell In[103], line 5, in integer_addition(*args)
      3 for i in args:
      4     if not isinstance(i, int):
----> 5         raise Exception(f"All provided types need to be <int>, you provided <{i.__class__.__name__}>.")
      6     out += i
      7 return out

Exception: All provided types need to be <int>, you provided <list>.

It would be clearer, if we substituted Exception with TypeError, to make it more clear, what exactly made our function stop: A wrong type.

Assert

The keyword assert allows us to save some space by putting the is expression and the raise of an Exception onto the same line.

Assert is often used for small checks and the Description (the part that gives more info about the Error) is often omitted.

def integer_addition(*args):
    out = 0
    for i in args:
        assert isinstance(i, int), f"All provided types need to be <class 'int'>, you provided <class '{i.__class__.__name__}'>."
        out += i
    return out

integer_addition(1, 2, 3, 4, 5, ['this is a list'])

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[104], line 8
      5         out += i
      6     return out
----> 8 integer_addition(1, 2, 3, 4, 5, ['this is a list'])

Cell In[104], line 4, in integer_addition(*args)
      2 out = 0
      3 for i in args:
----> 4     assert isinstance(i, int), f"All provided types need to be <class 'int'>, you provided <class '{i.__class__.__name__}'>."
      5     out += i
      6 return out

AssertionError: All provided types need to be <class 'int'>, you provided <class 'list'>.

Exception handling

There are some keywords that help you manage Exceptions and Errors. The general flow looks like this:

try:
    pass
except Exception:
    pass
else:
    pass
finally:
    pass

That’s a lot of keywords. Let’s go through them in order. Let’s use the integer_addition() function.

numbers = [1, 2, 3, 4.5, 5, 5.6]
try: out = integer_addition(*numbers)
except Exception:
    print('Dropping every non-integer values.')
    out = integer_addition(*filter(lambda x: isinstance(x, int), numbers))
print(out)

Dropping every non-integer values.
11

If we want to do something after the Exception is raised, but still stop the execution, we can simply input a raise

numbers = [1, 2, 3, 4.5, 5, 5.6]
try: out = integer_addition(*numbers)
except Exception:
    print('There are some non-integer values.')
    print([type(i) for i in numbers])
    raise

There are some non-integer values.
[<class 'int'>, <class 'int'>, <class 'int'>, <class 'float'>, <class 'int'>, <class 'float'>]

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[106], line 2
      1 numbers = [1, 2, 3, 4.5, 5, 5.6]
----> 2 try: out = integer_addition(*numbers)
      3 except Exception:
      4     print('There are some non-integer values.')

Cell In[104], line 4, in integer_addition(*args)
      2 out = 0
      3 for i in args:
----> 4     assert isinstance(i, int), f"All provided types need to be <class 'int'>, you provided <class '{i.__class__.__name__}'>."
      5     out += i
      6 return out

AssertionError: All provided types need to be <class 'int'>, you provided <class 'float'>.

Using the except Exception expression is oftentimes too vague, because it catches all Errors (it is after all the base class). It would be better to just catch the Error we’re after.

This is bad practice, because it does not catch the correct Exception.

numbers = [1, 2, 3, 5]
try:
    out = integer_addition(*numbers)
    var = bad_var
except Exception:
    print('There are some non-integer values.')
    print([type(i) for i in numbers])

There are some non-integer values.
[<class 'int'>, <class 'int'>, <class 'int'>, <class 'int'>]

With this code, which catches the integer_addition()’s AssertionError we see, that the var = bad_var statement is the one causing trouble.

numbers = [1, 2, 3, 5]
try:
    out = integer_addition(*numbers)
    var = bad_var
except AssertionError:
    print('There are some non-integer values.')
    print([type(i) for i in numbers])

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[108], line 4
      2 try:
      3     out = integer_addition(*numbers)
----> 4     var = bad_var
      5 except AssertionError:
      6     print('There are some non-integer values.')

NameError: name 'bad_var' is not defined

Wit the as keyword we can also save the Exception to a variable.

numbers = [1, 2, 3, 5]
try:
    out = integer_addition(*numbers)
    var = bad_var
except AssertionError:
    print('There are some non-integer values.')
    print([type(i) for i in numbers])
except NameError as e:
    print(e)
    print("Some variables are missing.")

name 'bad_var' is not defined
Some variables are missing.

Now let’s look into the very long exception handling from above:

The else clause is run, when the try clause does not raise an exception.

numbers = [1, 2, 3, 5]
bad_var = 1
try:
    out = integer_addition(*numbers)
    var = bad_var
except AssertionError:
    print('There are some non-integer values.')
    print([type(i) for i in numbers])
except NameError as e:
    print(e)
    print("Some variables are missing.")
else:
    print('Everything ran successfully.')
    print('The sum is: ', out)

# delete bad_var, so the cells above still raise a NameError
del bad_var

Everything ran successfully.
The sum is:  11

The fianlly clause runs no matter what happens. This can be used to free resources no matter what happens. If one of your objects occupies much space in the RAM (opened files, opened databases), you can use the finally as a sort-of garbage collection to free up the RAM again.

numbers = [1, 2, 3, 5]
bad_var = 1
try:
    out = integer_addition(*numbers)
    var = bad_var
except AssertionError:
    print('There are some non-integer values.')
    print([type(i) for i in numbers])
except NameError as e:
    print(e)
    print("Some variables are missing.")
else:
    print('Everything ran successfully.')
    print('The sum is: ', out)
finally:
    print('Executing Finally...')

# delete bad_var, so the cells above still raise a NameError
del bad_var

Everything ran successfully.
The sum is:  11
Executing Finally...

Custom Exceptions

You can also create your own custom Exceptions.

class BadError(Exception):
    """Raised when the Error is really bad."""
    def __init__(self, message):
        self.message = '(╯°□°)╯︵ ┻━┻ ' + message + ' ┬─┬ノ( ◕◡◕ ノ)'
        super().__init__(self.message)

raise BadError("Welp. That's all about Exceptions.")

---------------------------------------------------------------------------
BadError                                  Traceback (most recent call last)
Cell In[112], line 7
      4         self.message = '(╯°□°)╯︵ ┻━┻ ' + message + ' ┬─┬ノ( ◕◡◕ ノ)'
      5         super().__init__(self.message)
----> 7 raise BadError("Welp. That's all about Exceptions.")

BadError: (╯°□°)╯︵ ┻━┻ Welp. That's all about Exceptions. ┬─┬ノ( ◕◡◕ ノ)

Warnings

Wanrings can be imported from the builtin library warnings. Warnings can: - Print text to the output to inform the user (‘default’). - Turn matching warnings into Exceptions (‘error’). - Never print matching warnings (‘ignore’). - Always print matching warnings (‘always’). - Print the first occurrence of a warning inside a module (used to inform the user about upcoming deprecations) (‘module’). - Print only the first occurrence of matching warnings (‘once’).

Let’s get into it. To print a standard warning just do something like this:

Simple warning

import warnings
# resets warnings to default
warnings.resetwarnings()

def protected_division(x, y):
    if y == 0:
        warnings.warn("Protected division. Returning 0.")
        return 0
    else:
        return x / y

print(protected_division(1, 0))
print(protected_division(2, 0))
print(protected_division(3, 0))
print(protected_division(4, 0))

0
0
0
0

/tmp/ipykernel_5016/1203745237.py:7: UserWarning: Protected division. Returning 0.
  warnings.warn("Protected division. Returning 0.")

Turn consecutive warnings into Exceptions

import warnings
# resets warnings to default
warnings.resetwarnings()

warnings.filterwarnings('error')

def protected_division(x, y):
    if y == 0:
        warnings.warn("Protected division. Returning 0.")
        return 0
    else:
        return x / y

print(protected_division(1, 0))
print(protected_division(2, 0))
print(protected_division(3, 0))
print(protected_division(4, 0))

---------------------------------------------------------------------------
UserWarning                               Traceback (most recent call last)
Cell In[114], line 14
     11     else:
     12         return x / y
---> 14 print(protected_division(1, 0))
     15 print(protected_division(2, 0))
     16 print(protected_division(3, 0))

Cell In[114], line 9, in protected_division(x, y)
      7 def protected_division(x, y):
      8     if y == 0:
----> 9         warnings.warn("Protected division. Returning 0.")
     10         return 0
     11     else:

UserWarning: Protected division. Returning 0.