10 Jupyter Notebook Hacks Every AI & ML Expert Should Know

Jupyter Notebooks have revolutionized the way data scientists and machine learning researchers work. These interactive, web-based environments allow you to combine live code, equations, visualizations, and explanatory text in a single document, creating a cohesive and reproducible record of your work.

Since its introduction in 2014, the Jupyter Notebook has seen explosive growth and adoption. According to a recent study, over 4.7 million Jupyter Notebooks were publicly available on GitHub as of late 2020. Many famous data scientists and ML experts, such as Wes McKinney (creator of pandas), Thomas Wiecki (PyMC3 developer), and Jeremy Howard (fast.ai), use Jupyter Notebooks extensively and have praised their utility.

While Jupyter Notebooks are incredibly powerful out of the box, knowing some tips and tricks can help you get even more out of this tool and become a true power user. As an AI and ML expert, I‘ve spent countless hours working in Jupyter Notebooks and have discovered many helpful hacks along the way.

Here are 10 of my favorite Jupyter Notebook hacks, with concrete examples of how they can streamline your data science and ML workflows. Trust me, these tips will take your notebooks to the next level!

1. Inline Matplotlib Plots

Matplotlib is the workhorse data visualization library of the Python data science stack. By default, Matplotlib plots open in a new window, which can get annoying if you‘re rapidly iterating on a plot design. The %matplotlib inline magic command solves this by rendering plots directly inside the notebook.

Here‘s an example:

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0, 10, 100)
y = np.sin(x)

plt.figure(figsize=(8, 4))
plt.plot(x, y, ‘r-‘, linewidth=2, label=‘sin(x)‘)
plt.xlabel(‘x‘)
plt.ylabel(‘y‘)
plt.legend()
plt.grid()
plt.title("Simple Line Plot")
plt.show()

By using %matplotlib inline, the plot appears directly below the code cell that creates it. This keeps your plot and the code that generated it together, making for a cleaner and more understandable notebook.

According to the official Matplotlib documentation, "this is the recommended way to use Matplotlib in Jupyter Notebooks." Trust the experts and adopt this tip in your own notebooks!

2. Quick Documentation Lookup

When you‘re writing code in a Jupyter Notebook, you often need to look up documentation for functions, classes, or methods. Instead of Googling or searching through documentation websites, simply use the Shift-Tab shortcut.

Place your cursor inside a function or class name and press Shift-Tab. A tooltip will appear showing the signature and docstring for that object. Press Shift-Tab multiple times to cycle through increasingly verbose documentation.

For example, put your cursor inside np.random.rand and press Shift-Tab:

Without leaving your notebook, you can quickly see that np.random.rand creates an array of random values between 0 and 1, and accepts size parameters to determine the shape of the output array.

This trick works for any function, class, or method, including ones you define yourself! It‘s an incredibly handy way to jog your memory without breaking your workflow.

3. Running Shell Commands

Jupyter Notebooks provide a convenient way to execute shell commands directly from a code cell using the ! prefix. This is super useful for tasks like file management, package installation, or quick data preprocessing using command line tools.

For example, to check what datasets are available in the current directory:

!ls data

Or to install the latest version of TensorFlow using pip:

!pip install tensorflow --upgrade

You can run any shell command you would normally use in the terminal. The output is captured and displayed directly in the notebook:

This saves you from having to switch to a terminal window, run the command, then switch back to the notebook. It keeps everything neatly contained in one environment.

As a data scientist who frequently uses shell commands for data munging and preprocessing, I find this feature indispensable. It lets me quickly prototype a data processing pipeline entirely within my Jupyter Notebook.

4. Easy Code Profiling

When working with big datasets or complex ML models in Jupyter Notebooks, it‘s important to keep an eye on performance. The %time, %timeit, %%time, and %%timeit magic commands let you quickly measure the execution time of code snippets.

To time a single line of code, use %time:

%time some_slow_function() 

To time an entire code cell, use %%time:

%%time
for i in range(1000):
    some_slow_function(i)

If you want to average the execution time over multiple runs, use %timeit or %%timeit:

%timeit some_slow_function()

%%timeit
for i in range(1000): 
    some_slow_function(i)

The %timeit and %%timeit commands will repeat the operation several times to obtain more robust performance metrics:

In my experience, these profiling magic commands are incredibly helpful for identifying performance bottlenecks, especially in data preprocessing code or model inference code. A few well-placed %timeits can help you optimize your notebook‘s runtime.

5. Importing Code Snippets

Let‘s say you have some useful utility functions defined in an external Python script. Instead of rewriting those functions in your notebook, you can use the %load magic command to import the code directly:

%load utils.py

This will insert the contents of utils.py into a new code cell:

You can also %load code from a URL:

%load https://raw.githubusercontent.com/jakevdp/PythonDataScienceHandbook/master/notebooks/02.02-The-Basics-Of-NumPy-Arrays.ipynb

This trick is handy for bringing in code snippets from your other projects or from online resources. I often use it to quickly experiment with code from data science blogs or Stack Overflow answers.

6. Passing Variables Between Notebooks

When working on complex data science projects, I often split my work across multiple notebooks: one for data cleaning, one for feature engineering, one for model training, etc. The %store magic command makes it easy to pass variables between these notebooks.

To save a variable for later use, run:

data = pd.read_csv("cleaned_data.csv")
%store data

Then in another notebook:

%store -r data

And the data variable is restored, without having to rerun the data cleaning code!

The %store command serializes variables to disk, making them available to other notebooks. This is much more convenient than always having to rerun previous notebooks to obtain their outputs.

7. Interactive Widgets

Jupyter Notebooks can include interactive GUI widgets for exploring your code outputs. This is powered by the ipywidgets library, which lets you add sliders, text boxes, drop-down menus, and more to your notebooks.

For example, here‘s a simple demo of interactive widgets for a Matplotlib plot:

%matplotlib inline
from ipywidgets import interact
import numpy as np
import matplotlib.pyplot as plt

def plot_sine(freq):
    x = np.linspace(0, 2*np.pi, 500)
    y = np.sin(freq*x)
    plt.figure(figsize=(8,5))
    plt.plot(x, y)
    plt.ylim(-1.1, 1.1)
    plt.title(f‘Sine Wave with Frequency {freq} Hz‘)
    plt.show()

interact(plot_sine, freq=(0.1, 5, 0.1))

As you move the slider, the plot updates in real-time to reflect the new frequency value. This is a fantastic way to explore different parameter values and visualize their effect.

The ipywidgets library supports many types of GUI controls. In my AI/ML work, I often use widgets to interactively tune hyperparameters, visualize different layers of a neural network, or demonstrate an algorithm step-by-step. It‘s a powerful way to poke and prod at your code.

8. Custom Notebook CSS

By default, Jupyter Notebooks have a clean and functional appearance. But maybe you prefer a different color scheme, font size, or layout. With a few lines of CSS, you can customize your notebook‘s style to suit your preferences.

First, create a custom.css file in your Jupyter config directory (usually ~/.jupyter/custom). Then add your custom CSS rules:

/* Increase code font size */
.CodeMirror pre {
    font-size: 16pt !important;
}

/* Add a border around each cell */
div.cell {
    border: 2px solid #eee;
    border-radius: 5px;
    margin-top: 20px;
}

/* Change the selected cell background color */
div.cell.selected {
    border-color: #66BB6A;
}

/* Justify text cells by default */
.text_cell_render {
    text-align: justify;
}

With just a few tweaks, you can make your notebook much more visually appealing and tailored to your tastes. I like increasing the code font size for presentations, and adding colored borders makes it easier to visually separate cells.

You can find a lot more Jupyter CSS snippets on GitHub and Stack Overflow. Try a few and see what works for you!

9. Helpful Extensions

Beyond the core Jupyter Notebook, there are many community-contributed extensions that provide additional functionality. Here are a few of my favorites:

• jupyter_contrib_nbextensions: This pack includes a variety of extensions, including a code formatter, table of contents generator, and automatic notebook version control.
• jupyter-matplotlib: Adds interactive features to Matplotlib plots in notebooks, like zooming, panning, and exporting.
• jupyterlab-toc: Generates a table of contents sidebar in JupyterLab for easier navigation.
• jupyterlab-vim: Vim key bindings for JupyterLab cells and editor windows.
• lckr-jupyterlab-variableinspector: Variable inspector extension for examining currently defined variables and their values.

You can browse and install extensions directly within Jupyter Notebook by going to Edit > nbextensions config:

Or if you prefer the command line:

# Install an extension 
jupyter nbextension install <extension-name>

# Enable the extension
jupyter nbextension enable <extension-name>

In my experience, a few well-chosen extensions can make a big difference in your productivity. I rely heavily on the table of contents and variable inspector extensions. Try some out and see what works for your workflow!

10. Multicursor Editing

Here‘s a small but mighty tip: Jupyter Notebooks support multi-cursor editing for manipulating several parts of a cell simultaneously.

Hold down Alt (or Option on macOS) while clicking to add a new cursor. Now whatever you type appears in all cursor locations! This is incredibly useful for tasks like renaming variables, fixing typos, or refactoring code.

I can‘t tell you how much time this trick has saved me. It‘s a lot faster than find-and-replace or making the same edit over and over.

You can also use keyboard shortcuts to add cursors: Alt-Shift-Up/Down to add a cursor above/below the current one. And Esc or clicking merges cursors back into one. Try it out!

Bonus: JupyterLab

If you‘re a fan of Jupyter Notebooks, you need to check out JupyterLab. It‘s the next-generation interface for Jupyter, with a more powerful and flexible UI.

JupyterLab feels like a full-fledged IDE. You can arrange multiple notebooks side-by-side, drag files from a file browser, and take advantage of other IDE features like code consoles and contextual help.

I‘ve been using JupyterLab for my recent machine learning projects, and it‘s been a big productivity boost. The ability to see multiple notebooks and scripts at once is a game-changer. If you haven‘t tried JupyterLab yet, I highly recommend giving it a spin!

Conclusion

These are just a few of my favorite hacks, tips, and tricks for Jupyter Notebooks. Incorporating even one or two of them into your workflow can save you a lot of time and headache.

As data scientists, so much of our work happens in notebooks: data cleaning, exploration, model building, visualization, etc. By mastering our tools, we can work faster and more efficiently. That‘s why I believe it‘s worth investing time to learn the ins and outs of Jupyter.

A quick personal story: During a recent ML project at work, I was tasked with building an image classification model. I used nearly all of the Jupyter tricks covered in this article: inline plots for data exploration, %timeit for profiling different model architectures, widgets to visualize layers and filters, multi-cursors to refactor code, and more. The end result was a high-performing model and an interactive Jupyter Notebook demo for the end users.

Since then, several colleagues have asked me for Jupyter tips, and I‘m always happy to evangelize the tools that help me work better. That‘s why I wrote this article—to share this knowledge more widely.

I hope these tips are as useful for you as they have been for me. Pick one and try it in your next notebook. I bet you‘ll be hooked! And if you have your own favorite Jupyter hacks, let me know in the comments.

Happy coding, and may your notebooks be clean, fast, and reproducible!