Magic Commands for Profiling in Jupyter Notebook

Jupyter Notebook is a versatile tool or IDE widely used mostly by data scientists, researchers, and programmers for interactive computing and data analysis, dashboards, and visualizations. It offers a unique and rich set of features. Some topics are there it can perform such as: running the code in different languages, timing the code execution, debugging the code, and profiling. We’ll discuss the commands that work like magic for profiling in Jupyter Notebook. Magic Commands are the special commands in Jupyter Notebook that either start with the ‘% ‘ or ‘%% ‘ sign and perform different operations and tasks. Profiling helps in identifying bottlenecks, optimizing code, and improving overall performance. So, let’s explore the world of magic commands in Jupyter Notebook, specifically designed for profiling our Python code. Before we jump into it, we first need to understand some of the key concepts that we should know about profiling in Jupyter Notebook. So, what actually profiling is in Jupyter Notebook?

What are Magic Commands?

Magic commands are special commands in Jupyter Notebook, and they are denoted with either a single % or %%. Here the commands with % are for line-level commands. The commands with %% are for cell-level commands or multi-line level code. These magic commands help us to perform a wide range of tasks that go beyond standard Python capabilities. Magic commands are like shortcuts that can make complex operations simple and easy to understand, they make our notebook more productive and efficient.

Type of Magic Commands

Magic commands in Jupyter Notebook are divided into two categories i.e. Line Magic Commands and Cell Magic Commands. By the names, we can understand that the line magic commands will start with % and the cell magic commands will start with %% as we’ve already discussed before as well.

Line Magic Commands: These are used to operate on a single line of code. For example: %timeit, %memit, %load, %reset, %who and etc.

Cell Magic Commands: These affect the entire cell. For example: %%time, %%writefile, %%html, %%latex, %%bash and etc.

How to Use the Magic Commands

Making use of the magic commands in Jupyter Notebook is quite straightforward. We just simply prefix our code with the appropriate magic command, and Jupyter Notebook takes care of the rest. These commands can be utilized for a bunch of different tasks, such as measuring code execution time, profiling memory usage, debugging, and more.

What is Profiling and Its Type

Profiling refers to the process of analyzing code performance and resource utilization in the programming world. There are mainly two types of profiling:

Code Profiling: Code profiling involves measuring the execution time of various parts of our code to identify bottlenecks or areas where optimization is needed.

Memory Profiling: Memory profiling focuses on tracking how our code uses system memory. It also helps in identifying memory leaks and inefficient memory usage patterns.

Basically, Profiling is the process of measuring the performance of code, such as how long does it take to run and how much memory does it use! If we want to profile a single line of code, we can use the ‘%prun‘ magic command. But if we want to profile a whole cell of code, we can use the ‘%%prun‘ magic command. Another magic command is ‘%lprun‘ and this is used for profiling the code line-by-line. ‘%lprun‘ is mostly used for pinpointing specific lines of code, if it is causing performace issues.

Profiling Jupyter Notebook

Now let’s try to understand what are the steps needed to profile in Jupyter Notebook.

1. Installing the required libraries:

First we need to ensure that we have the necessary libraries installed in out system. As we need ‘line_profiler‘ for ‘%lprun’ and ‘cProfile‘ for ‘%prun‘. If it’s not installed already in our notebook, we can install them by using the following commad:

!pip install line_profiler

2. Load the Magic Commands

In our Jupyter Notebook, we then need to load the ‘%lprun’ and ‘%prun’ magic commands. We’ll now use the following commands to load them:

3. Profiling the Code

Now we’ll use ‘%lprun’ to profile specific lines of code and ‘%prun’ to profile entire functions as we have dicussed about this in the introduction. Let’s understand this with a small example:

Output:

Timer unit: 1e-09 s
Total time: 0.00622773 s
File: /tmp/ipykernel_601603/4220250964.py
Function: fib at line 3
Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     3                                           def fib(n):
     4     21891    3171786.0    144.9     50.9      if n < 2:
     5     10946    1101316.0    100.6     17.7          return n
     6                                               else:
     7     10945    1954627.0    178.6     31.4          return fib(n-1) + fib(n-2)

When you run the above code, your output will be telling you about; Line, Hits, Time, Per Hit, % Time, and Line Contents. It’ll be somewhat looking like this:

Examples

There are many magic commands that are useful for different scenarios. Here we will explore some of the commands that are dedicated to profiling.

Line-by-Line Profiling (%lprun)

Suppose we have a function ‘calculate_sum’ that adds all the elements of a list. Then we suspect that there’s a performance issue, and we want to profile it. Let’s see how we can use %lprun:

Output:

Timer unit: 1e-07 s

Total time: 2.4e-06 s
File: C:\Users\GFG0265\AppData\Local\Temp\ipykernel_32000\3964288041.py
Function: calculate_sum at line 1

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           def calculate_sum(my_list):
     2         1         24.0     24.0    100.0      return sum(my_list)

Function-Level Profiling (%prun)

Imagine we have a complex function process_data that we want to profile. Then for that we can use %prun like this:

Output:

 4 function calls in 0.000 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 2018323565.py:1(process_data)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Profiling a Complex Script

Suppose we have a complex script that involves multiple functions, and we want to identify which functions are consuming the most time during execution. Let’s try to understand it with the complete code example and the output format:

Output:

Result 1: 499999500000
Result 2: 499999500000

To profile the performance of the Python function “main()”, the code employs the %prun magic command with the -s cumulative option. This generates a report that identifies the sections of the code that consume the most cumulative time, providing insights for optimizing the code and improving its performance.

Output:

81 function calls in 0.138 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.138    0.138 {built-in method builtins.exec}
        1    0.000    0.000    0.138    0.138 <string>:1(<module>)
        1    0.000    0.000    0.138    0.138 2450259892.py:10(main)
        1    0.098    0.098    0.098    0.098 2450259892.py:1(slow_function)
        1    0.000    0.000    0.039    0.039 2450259892.py:7(fast_function)
        1    0.038    0.038    0.038    0.038 {built-in method builtins.sum}
        2    0.000    0.000    0.001    0.001 {built-in method builtins.print}
        8    0.000    0.000    0.001    0.000 iostream.py:610(write)
        8    0.001    0.000    0.001    0.000 iostream.py:532(_schedule_flush)
        1    0.000    0.000    0.000    0.000 iostream.py:243(schedule)
        1    0.000    0.000    0.000    0.000 socket.py:543(send)
        8    0.000    0.000    0.000    0.000 iostream.py:505(_is_master_process)
        1    0.000    0.000    0.000    0.000 threading.py:1185(is_alive)
        1    0.000    0.000    0.000    0.000 threading.py:1118(_wait_for_tstate_lock)
        8    0.000    0.000    0.000    0.000 {method 'write' of '_io.StringIO' objects}
        8    0.000    0.000    0.000    0.000 {built-in method nt.getpid}
        1    0.000    0.000    0.000    0.000 {method 'acquire' of '_thread.lock' objects}
        8    0.000    0.000    0.000    0.000 {method '__exit__' of '_thread.RLock' objects}
        1    0.000    0.000    0.000    0.000 iostream.py:127(_event_pipe)
        8    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
        8    0.000    0.000    0.000    0.000 {built-in method builtins.len}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {method 'append' of 'collections.deque' objects}
        1    0.000    0.000    0.000    0.000 threading.py:568(is_set

Code Timing (%timeit )

The %timeit magic command is used to measure the execution of the time of a single line of code (as we have already discussed about the single line and a cell of code). It runs the specified code multiple times and provides us with information about the average time taken.

Output:

8.52 µs ± 21.5 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

%memit for Memory Profiling

The %memit magic command allows us to profile memory usage for a single line of code. It’s particularly helpful in identifying memory-intensive operations.

Step 1: We need to Install the memory_profiler package if we haven’t already.

!pip install memory-profiler

It’ll install the memory-profiler. After installing we’re not done yet for using %%memit.

Step 2: After installing memory-profiler we need to load its extension in our Jupyter Notebook now.

And now we’ve succesfully loaded its extension. So, let’s use it now.

Output:

peak memory: 70.90 MiB, increment: 0.32 MiB

Table for Magic commands

Conclusion

Magic commands in Jupyter Notebook make profiling our code a breeze in Python. Whether we need to analyze individual lines or entire functions, %lprun and %prun, %timeit, %memit, %%time, %%writefile, %%html, %%latex, %%bash and more; there provide valuable insights to help us optimize our code for better performance or for any complex operations. Profiling is an essential skill for any programmer or data scientist or an seasoned enthusiast, and with these magic commands, we can streamline the process and become a more efficient coder/programmer.