A Few Optimization Techniques in Python #1

Posted in July 27, 2020 by

Categories: Computer Science Programming Python

Tags: , , ,

Reading Time: 7 minutes

This blog post is about a few techniques and ways of writing Python code that will make it much faster in execution. Optimization of code is very important but I am not a Python expert, yet ๐Ÿ˜‹ . I constantly learning that beautiful language. By doing that I have learned a few ways to make written code faster.

If you like to learn mathematics and programming in Python I can recommend my previous post about a few simple mathematical implementations in that language. Feel free to check it out but I have to warn you that it is written in Polish.

If you are a beginner at programming I can recommend you to read one of the blogs I read about IT. It is blog of Mateusz Rus. I recommend you to read this specific post about “How to Learn Programming” but it is also in Polish.

Storing Code Already Compiled

In Python, we can use one of three build-in functions do evaluate code stored in simple String data type. These functions are:

  • eval()
  • exec()
  • compile()

With those functions, we can execute code that comes from a database, websites, or any other place when is stored in String data format. Well not exactly ๐Ÿง . Eval() function is little less powerful than exec() but still workable.

Our Function

To be able to speed up your code we will try to understand compile() function. I take three arguments.

sourceCompiled = compile(source, filename, "exec | eval")

The first argument, “source“, is our code in String data format. The second “filename” is a path to a file where our source code is stored. In the last argument, the mode is one of the modes in which our compile function works.

When it is set to “exec” like it will be in our example we can process any code we want, compile it and store its compiled version in a variable called sourceCompiled.

When it is set to “eval” it will evaluate our expression which could be like:

x + 1

Our Example

In our example lets imagine that after some changes in our code we want to add to variable reportLine number 1 like:

reportLine += 1

If we would make 1000 changes in code above instruction would execute 1000 times ๐Ÿ˜” . Let’s assume now that we will store that line of code in a String data type and run it with exec function.

We want to calculate how much time it will take to execute 1000 times code like below:

import time

source = "reportLine += 1"
reportLine = 0

start = time.time()
for _ in range(1000):
    exec(source)
stop = time.time()

time_passed = stop - start

print("1000 Executions of code that was not precompiled {}".format(time_passed))

So when we don’t precompile reportLine += 1 instruction it takes approximately 0.014 of a second to do 1000 additions. Our reportLine variable should store now number 1000.

Better Solution

Lets check out now how much time it will take to execute the same code but with precompiled instruction reportLine += 1.

import time

source = "reportLine += 1"
reportLine = 0

sourceCompiled = compile(source, "some file", "exec")
start = time.time()
for _ in range(1000):
    exec(sourceCompiled)
stop = time.time()

time_passed = stop - start

print("1000 Executions of code that was precompiled {}".format(time_passed))

As we see precompiled code execution is much faster. It took approximately 0.0005 of a second to do a 1000 additions in instruction reportLine += 1.

If we would like to calculate how much faster is that code we can make a simple mathematical comparison:


(1)   \begin{equation*} 0.0005x = 0.014 \end{equation*}

(2)   \begin{equation*}x = \frac{0.014} {0.0005} \end{equation*}

(3)   \begin{equation*} x = 28 \end{equation*}

We calculated that using precompiled code is approximately 28 times faster ๐Ÿ‘ ๐Ÿ˜ in execution than using code that is not precompiled with exec function.

That would be the first optimization technique in Python in this article. Let’s jump to second one.

Optimization of Function Using Cache

When we use a function that always returns fixed values, deterministic functions ๐Ÿคจ , we can facilitate cache. It will remember the returned values from our function and use them in our computation tasks.

Without Cache

Let’s assume we have to calculate the factorial of a given set of numbers. It can be from 1 to 10. So we could have a function which calculates factorial from a given number like below:

import time

def factorial(n):

    time.sleep(0.1)
    if n == 1:
        return 1
    else:
        return n * factorial(n - 1)

This deterministic function return always the same number for the same passed argument. For a number 5, the result will be 120, for 4 it will be 24, and so on.

In this function, we use the sleep function of the time module to show the result of our optimization in a better way.

Now we can calculate how much it will take time to calculate a factorial for every number from 1 to 10:

start = time.time()

for i in range(1, 11):
    print("{}! = {}".format(i, factorial(i)))

stop = time.time()

result = stop - start
print("Execution time of above code is {}.".format(result))

As we see on below screenshot the execution time is about 5.7 second.

With Cache

Now we could think that it would be much better to store in cache every previous result of the execution of our factorial function. We couldn’t be more correct ๐Ÿ˜‰ .

We import the module functools to add to our factorial function a decorator. It adds a store to cache capabilities to this function. The decorator’s name is @functools.lru_cache().

import time
import functools

@functools.lru_cache()
def factorial(n):

    time.sleep(0.1)
    if n == 1:
        return 1
    else:
        return n * factorial(n - 1)

With added decorator from functools module. The factorial function will be able to remember previous execution results and use it in future computations.

For example, when factorial function computes factorial from number 5 it doesn’t compute factorial from numbers 4, 3, 2, and 1 because previously it was already computed.

Our function just takes already computed results. These are stored in the cache and use it to compute factorial from number 5.

start = time.time()

for i in range(1, 11):
    print("{}! = {}".format(i, factorial(i)))

stop = time.time()

result = stop - start
print("Execution time of above code after optimisation is {}.".format(result))

As we see below the execution time is more than 5 times faster ๐Ÿ‘Œ ๐Ÿค“ :

Second Look

Additionally when we will execute the same instructions again the computation time will be close to zero. It is because all computation results are already in the cache. Computer doesn’t have to compute it anymore. Let’s see.

import time
import functools


@functools.lru_cache()
def factorial(n):

    time.sleep(0.1)
    if n == 1:
        return 1
    else:
        return n * factorial(n - 1)


start = time.time()

for i in range(1, 11):
    print("{}! = {}".format(i, factorial(i)))

stop = time.time()

result = stop - start
print("Execution time of above code after optimisation is {}.".format(result))

print("*" * 30)

start = time.time()

for i in range(1, 11):
    print("{}! = {}".format(i, factorial(i)))

stop = time.time()

result = stop - start
print("Execution time of above code after optimisation is {}.".format(result))

Hurray ๐Ÿ˜Š ! As we see below execution time of function factorial in the second loop is close to nothing.

If we would like to check how our optimization function goes with decorator we can use:

factorial.cache_info()

It tells us how much data we have saved in the cache (10). How many results we can store in the cache (default is 128). The times computer couldn’t find the result in the cache (10). How many times cache was used to speed up execution time.

The last number is 9 because when we run this function after the first loop the last computation would be factorial from number 10. If we would run this function after the second loop it would be hits=19 because we would store hits from the first loop (9) and second loop (10).

To Sum Up

In this post, I tried to explain a few techniques to optimize your code in Python. I think I will write a few more posts like this in the future when my knowledge of Python will grow.

If you would like to learn more about Python and know the Polish language I can recommend you one of the Polish blogs. I read it quite frequently.

It is a blog by Mateusz Mazurek. I recommend this series of posts about “What not to do in Python?”.

I plan to write my own project in Python soon. It will be a simple game using the Turtle module. I am planning it from a long time now but didn’t have time to start. I will try to start that project soon and write here how it is going plus I will put everything on Github ๐Ÿค“ .

If you like this post don’t hesitate to comment it, like it or share it. I would appreciate any insights in the comment section ๐Ÿค” .

Have a great day and see you soon. Bye ๐Ÿ‘‹ .



Leave a Reply

Your email address will not be published. Required fields are marked *

1 + 7 =