Understanding Python decorators

Recently, I found that my understanding of Python decorators wasn't as sound as I wanted it to be. I hate finding out that my understanding isn't sound! But it meant I spent a bit of time making sure I understood, and I wanted to share my thoughts and learnings here :)

What does the @ syntax actually do?

Python decorators allow us to neatly modify functions and classes.
Syntactically, this is achieved with the @ symbol.

Let's say we have some decorator we've defined, called dec. We can apply this decorator to our function func like so:

def func(arg1, arg2, ...):

This is just sugar for:

def func(arg1, arg2, ...):
func = dec(func)
  1. This explains why we end up with so much nesting for taking arguments in the decorator:

The @ syntax will pass the function object into whatever comes after the @ sign. Above, dec is expecting a function object, and func gets passed to it. func gets replaced with dec(func). It’s all good.

But what if we want to take an argument in the decorator?

def tags(tag_name): def tags_decorator(func): def func_wrapper(name): return "<{0}>{1}".format(tag_name, func(name)) return func_wrapper return tags_decorator

@tags("p") def get_text(name): return "Hello "+name

The decorated function is equivalent to:

get_text = tags("p")(get_text)

As tags returns a decorator. The @ is behaving just as before: it takes whatever comes after the @ (in this case tags(“p”)), and passes get_text in.

(example from https://www.thecodeship.com/patterns/guide-to-python-function-decorators/)

  1. This nesting is ugly, how do we reduce it?

A lot of people use classes as decorators by overriding the call method (see https://stackoverflow.com/a/9663601 here if you need an explanation of that method). You can see Jan did this in the task_utils class retry. Nesting is reduced because we can store stuff as instance variables.

class retry:... # see task_utils.py

@retry(times=5) def func()...

Again, the @ syntax works the same as before

retry(times=5) returns an instance of the class, to which func is passed. When we called the instance, the call method is called.

So the above is equivalent to:
func = retry(times=5)(func)

  1. Ok but can I reduce triple def nesting and avoid classes? (The original question that spawned all of this)

The decorator function (in Jan’s example, the call method of the class, or in the get_text example, tags_decorator) has to return a function, otherwise we’d be replacing our decorated function with something not callable.

With only optional arguments being passed to the decorator, yes! Please see cmr.common.retry, which I wrote. (probably best ot have it open alongside this email)

def retry(func=None, exceptions=(Exception,), retries=5): ...

Either we do: @retry def function(...

in which case the func keyword argument is None, so we return the functools.partial callable object 😊

Or we do: @retry(retries=5) def function(...

in which case we get, via @ sign magic:

function = functools.partial(retry, exceptions=exceptions, retries=retries)(function)

pretty neat actually…let me know if this doesn’t make sense and we can talk through it.

I haven’t quite figured out if you could do this neatly with positional arguments, and I’m dying to eat and to pee. Let’s think about it together 😊