A Beginners Guide to Multi-Processing in Python

Introduction

In today’s era of Big Data, Python has emerged as a highly coveted programming language. In this article, we’ll delve into a vital aspect that empowers Python’s prowess – Multi-Processing. This feature distinguishes Python as a potent tool, enabling efficient handling of complex tasks by harnessing the full potential of modern hardware.

Now before we dive into the nitty-gritty of Multi-Processing, I suggest you read my previous article on Threading in Python, since it can provide a better context for the current article.

This article was published as a part of the Data Science Blogathon.

What is Multi-Processing?

Let us say you are an elementary school student who is given the mind-numbing task of multiplying 1200 pairs of numbers as your homework. Let us say you are capable of multiplying a pair of numbers within 3 seconds. Then on a total, it takes 1200*3 = 3600 seconds, which is 1 hour to solve the entire assignment.  But you have to catch up on your favorite TV show in 20 minutes.

What would you do? An intelligent student, though dishonest, will call up three more friends who have similar capacity and divide the assignment. So you’ll get 250 multiplications tasks on your plate, which you’ll complete in 250*3 = 750 seconds, that is 15 minutes. Thus, you along with your 3 other friends, will finish the task in 15 minutes, giving you 5 minutes time to grab a snack and sit for your TV show. The task just took 15 minutes when 4 of you work together, which otherwise would have taken 1 hour.

This is the basic ideology of Multi-Processing. If you have an algorithm that can be divided into different workers(processors), then you can speed up the program. Machines nowadays come with 4,8 and 16 cores, which then can be deployed in parallel.

Multi-Processing in Data Science?

Multi-Processing has two crucial applications in Data Science.

Input-Output Processes

Any data-intensive pipeline has input, output processes where millions of bytes of data flow throughout the system. Generally, the data reading(input) process won’t take much time but the process of writing data to Data Warehouses takes significant time. The writing process can be made in parallel, saving a huge amount of time.

Training Models

Though not all models can be trained in parallel, few models have inherent characteristics that allow them to get trained using parallel processing. For example, the Random Forest algorithm deploys multiple Decision trees to take a cumulative decision. These trees can be constructed in parallel. In fact, the sklearn API comes with a parameter called n_jobs, which provides an option to use multiple workers.

Multi-Processing in Python using Process Class

Now let us get our hands on the multiprocessing library in Python.

Multiprocessing Library in Python

Take a look at the following code

Python Code:

The above code is simple. The function sleepy_man sleeps for a second and we call the function two times. We record the time taken for the two function calls and print the results. The output is as shown below:

Starting to sleep
Done sleeping
Starting to sleep
Done sleeping
Done in 2.0037 seconds

This is expected as we call the function twice and record the time. The flow is shown in the diagram below.

Incorporate Multi-Processing into the Code

Now let us incorporate Multi-Processing into the code

import multiprocessing
import time
def sleepy_man():
    print('Starting to sleep')
    time.sleep(1)
    print('Done sleeping')

tic = time.time()
p1 =  multiprocessing.Process(target= sleepy_man)
p2 =  multiprocessing.Process(target= sleepy_man)
p1.start()
p2.start()
toc = time.time()

print('Done in {:.4f} seconds'.format(toc-tic))

Here multiprocessing.Process(target= sleepy_man) defines a multi-process instance. We pass the required function to be executed, sleepy_man, as an argument. We trigger the two instances by p1.start().

The output is as follows:

Done in 0.0023 seconds
Starting to sleep
Starting to sleep
Done sleeping
Done sleeping

Now notice one thing. The time log print statement got executed first. This is because along with the multi-process instances triggered for the sleepy_man function, the main code of the function got executed separately in parallel. The flow diagram given below will make things clear.

Executing Join() Function

In order to execute the rest of the program after the multi-process functions are executed, we need to execute the function join().

import multiprocessing
import time

def sleepy_man():
    print('Starting to sleep')
    time.sleep(1)
    print('Done sleeping')

tic = time.time()
p1 =  multiprocessing.Process(target= sleepy_man)
p2 =  multiprocessing.Process(target= sleepy_man)
p1.start()
p2.start()
p1.join()
p2.join()
toc = time.time()

print('Done in {:.4f} seconds'.format(toc-tic))

Now the rest of the code block will only get executed after the multiprocessing tasks are done. The output is shown below.

Starting to sleep
Starting to sleep
Done sleeping
Done sleeping
Done in 1.0090 seconds

The flow diagram is shown below.

Multiprocessing Using a For Loop

Since the two sleep functions are executed in parallel, the function together takes around 1 second.

We can define any number of multi-processing instances. Look at the code below. It defines 10 different multi-processing instances using a for loop.

import multiprocessing
import time

def sleepy_man():
    print('Starting to sleep')
    time.sleep(1)
    print('Done sleeping')

tic = time.time()

process_list = []
for i in range(10):
    p =  multiprocessing.Process(target= sleepy_man)
    p.start()
    process_list.append(p)

for process in process_list:
    process.join()

toc = time.time()

print('Done in {:.4f} seconds'.format(toc-tic))
The output for the above code is as shown below.

Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done in 1.0117 seconds

Here the ten function executions are processed in parallel and thus the entire program takes just one second. Now my machine doesn’t have 10 processors. When we define more processes than our machine, the multiprocessing library has a logic to schedule the jobs. So you don’t have to worry about it.

import multiprocessing
import time

def sleepy_man(sec):
    print('Starting to sleep')
    time.sleep(sec)
    print('Done sleeping')

tic = time.time()

process_list = []
for i in range(10):
    p =  multiprocessing.Process(target= sleepy_man, args = [2])
    p.start()
    process_list.append(p)

for process in process_list:
    process.join()

toc = time.time()

print('Done in {:.4f} seconds'.format(toc-tic))
The output for the above code is as shown below.

Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Starting to sleep
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done sleeping
Done in 2.0161 seconds

Since we passed an argument, the sleepy_man function slept for 2 seconds instead of 1 second.

Multi-Processing in Python using Pool Class

In the last code snippet, we executed 10 different processes using a for a loop. Instead of that we can use the Pool method to do the same.

import multiprocessing
import time

def sleepy_man(sec):
    print('Starting to sleep for {} seconds'.format(sec))
    time.sleep(sec)
    print('Done sleeping for {} seconds'.format(sec))

tic = time.time()

pool = multiprocessing.Pool(5)
pool.map(sleepy_man, range(1,11))
pool.close()

toc = time.time()

print('Done in {:.4f} seconds'.format(toc-tic))

multiprocessing.Pool(5) defines the number of workers. Here we define the number to be 5. pool.map() is the method that triggers the function execution. We call pool.map(sleepy_man, range(1,11)). Here, sleepy_man  is the function that will be called with the parameters for the functions executions defined by range(1,11)  (generally a list is passed). The output is as follows:

Starting to sleep for 1 seconds
Starting to sleep for 2 seconds
Starting to sleep for 3 seconds
Starting to sleep for 4 seconds
Starting to sleep for 5 seconds
Done sleeping for 1 seconds
Starting to sleep for 6 seconds
Done sleeping for 2 seconds
Starting to sleep for 7 seconds
Done sleeping for 3 seconds
Starting to sleep for 8 seconds
Done sleeping for 4 seconds
Starting to sleep for 9 seconds
Done sleeping for 5 seconds
Starting to sleep for 10 seconds
Done sleeping for 6 seconds
Done sleeping for 7 seconds
Done sleeping for 8 seconds
Done sleeping for 9 seconds
Done sleeping for 10 seconds
Done in 15.0210 seconds

Pool class is a  better way to deploy Multi-Processing because it distributes the tasks to available processors using the First In First Out schedule. It is almost similar to the map-reduce architecture- in essence, it maps the input to different processors and collects the output from all processors as a list. The processes in execution are stored in memory and other non-executing processes are stored out of memory.

Whereas in Process class, all the processes are executed in memory and scheduled execution using FIFO policy.

Comparing the Time Performance for Calculating Perfect Numbers

Hitherto, we played around with multiprocessing functions on sleep functions. Now let us take a function that checks if a number is a Perfect Number or not. For those who don’t know, A number is a perfect number if the sum of its positive divisors is equal to the number itself. We will be listing the Perfect numbers less than or equal to 100000. We will implement it in 3 ways- Using a regular for loop, using multiprocess.Process() and multiprocess.Pool().

Using Regular For Loop

import time

def is_perfect(n):
    sum_factors = 0
    for i in range(1, n):
        if (n % i == 0):
            sum_factors = sum_factors + i
    if (sum_factors == n):
        print('{} is a Perfect number'.format(n))

tic = time.time()
for n in range(1,100000):
    is_perfect(n)
toc = time.time()

print('Done in {:.4f} seconds'.format(toc-tic))
The output for the above program is shown below.

6 is a Perfect number
28 is a Perfect number
496 is a Perfect number
8128 is a Perfect number
Done in 258.8744 seconds

Using a Process Class

import time
import multiprocessing

def is_perfect(n):
    sum_factors = 0
    for i in range(1, n):
        if(n % i == 0):
            sum_factors = sum_factors + i
    if (sum_factors == n):
        print('{} is a Perfect number'.format(n))

tic = time.time()

processes = []
for i in range(1,100000):
    p = multiprocessing.Process(target=is_perfect, args=(i,))
    processes.append(p)
    p.start()

for process in processes:
    process.join()

toc = time.time()
print('Done in {:.4f} seconds'.format(toc-tic))

The output for the above program is shown below

6 is a Perfect number
28 is a Perfect number
496 is a Perfect number
8128 is a Perfect number
Done in 143.5928 seconds

As you could see, we achieved a 44.4% reduction in time when we deployed Multi-Processing using Process class, instead of a regular for loop.

Using a Pool Class

import time
import multiprocessing

def is_perfect(n):
    sum_factors = 0
    for i in range(1, n):
        if(n % i == 0):
            sum_factors = sum_factors + i
    if (sum_factors == n):
        print('{} is a Perfect number'.format(n))

tic = time.time()
pool = multiprocessing.Pool()
pool.map(is_perfect, range(1,100000))
pool.close()
toc = time.time()

print('Done in {:.4f} seconds'.format(toc-tic))

The output for the above program is shown below.

6 is a Perfect number
28 is a Perfect number
496 is a Perfect number
8128 is a Perfect number
Done in 74.2217 seconds

As you could see, compared to a regular for loop we achieved a 71.3% reduction in computation time, and compared to the Process class, we achieve a 48.4% reduction in computation time.

Thus, it is very well evident that by deploying a suitable method from the multiprocessing library, we can achieve a significant reduction in computation time.

Conclusion

In conclusion, mastering multi-processing in Python is a significant step for anyone venturing into the world of data-driven applications and performance optimization. With the ability to leverage multiple CPU cores, Python’s multi-processing empowers developers to tackle Big Data challenges, parallelize tasks, and enhance program efficiency. By understanding its principles, utilizing the multiprocessing library effectively, and optimizing code for concurrency, developers can unlock Python’s full potential. As technology evolves and data volumes continue to grow, the skill of multi-processing becomes increasingly invaluable, ensuring Python remains a top choice for tackling the most demanding computational tasks and maintaining its status as a programming powerhouse.

Frequently Asked Questions

The media shown in this article are not owned by Analytics Vidhya and is used at the Author’s discretion. 

Karthik Sundar

28 Sep 2023