Optimizing Python Performance for AI and ML with timeit

In the world of artificial intelligence (AI) and machine learning (ML), performance is paramount. Training sophisticated ML models like deep neural networks can take hours, days, or even weeks, so every bit of efficiency you can squeeze out of your code counts. The Python standard library provides a useful tool called timeit for measuring the execution time of code snippets and comparing different implementations. In this article, we‘ll dive into how to effectively use timeit to optimize AI and ML code, with a focus on best practices and analysis relevant to AI/ML workloads.

Why Performance Matters for AI/ML

Before we jump into the details of timeit, it‘s worth discussing why performance is so critical for AI/ML tasks. There are a few key reasons:

1. Training time: Most ML models train through an iterative optimization process that can require a huge number of iterations through the training data. Shaving even a few milliseconds off the time for each training step can add up to massive time savings over a full training run.

2. Inference latency: In many real-world applications, ML models need to generate predictions in real-time in response to user input or streaming data. Faster inference means a better user experience and enables new applications.

3. Resource efficiency: Training large ML models requires a lot of computational resources, which can be expensive. More efficient code means you can train bigger models with the same resources, or the same models with less expensive hardware.

4. Iteration speed: The process of developing ML models is highly experimental, often requiring many iterations to tune model architectures and hyperparameters. Faster code allows researchers and engineers to iterate faster and discover better models.

With this context in mind, let‘s see how timeit can help us write faster AI/ML code in Python.

Basic Usage of timeit

The core functionality of timeit is the timeit.timeit() function, which takes a Python statement to measure, along with a few optional parameters:

import timeit

# Measure the time to create a list of squares
timeit.timeit(‘[x**2 for x in range(1000)]‘, number=1000)

This will measure the execution time of the list comprehension statement 1000 times and return the cumulative time in seconds. You can also provide setup code that‘s run once before the main statement:

import timeit

setup = ‘‘‘
import numpy as np
arr = np.random.rand(1000)
‘‘‘

timeit.timeit(‘np.dot(arr, arr)‘, setup=setup, number=1000)

This measures the time for 1000 executions of a dot product of a NumPy array with itself, after first creating the random array in the setup code. NumPy is a critical library for numerical computing in Python, so measuring NumPy operations is common in AI/ML code.

Comparing Implementations with timeit

One of the most powerful applications of timeit is comparing the performance of different implementations of an operation. This is especially useful in AI/ML development, where there are often multiple libraries available for key tasks and a variety of ways to implement a given model architecture.

Let‘s look at a concrete example. Say we want to compare the inference speed of a basic feedforward neural network implemented with two popular ML frameworks – PyTorch and TensorFlow. We can use timeit to measure the forward pass time for a batch of data with each framework:

import numpy as np
import timeit

# Generate random input data
batch_size = 64
input_size = 1024
x = np.random.randn(batch_size, input_size).astype(np.float32)

# PyTorch implementation
pytorch_setup = ‘‘‘
import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

net = Net(input_size, 512, 10)
‘‘‘

pytorch_stmt = ‘‘‘
torch_x = torch.from_numpy(x)
net(torch_x)
‘‘‘

# TensorFlow implementation  
tensorflow_setup = ‘‘‘
import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Dense(512, activation=‘relu‘, input_shape=(input_size,)),
    tf.keras.layers.Dense(10)
])
‘‘‘

tensorflow_stmt = ‘‘‘
tf_x = tf.convert_to_tensor(x)
model(tf_x)
‘‘‘

num_iters = 1000
pytorch_time = timeit.timeit(stmt=pytorch_stmt, setup=pytorch_setup, number=num_iters)
tensorflow_time = timeit.timeit(stmt=tensorflow_stmt, setup=tensorflow_setup, number=num_iters)

print(f"PyTorch time: {pytorch_time:.5f} seconds")
print(f"TensorFlow time: {tensorflow_time:.5f} seconds")

On my machine with an NVIDIA GTX 1080 Ti GPU, this prints:

PyTorch time: 0.34867 seconds
TensorFlow time: 1.55275 seconds

Indicating that for this simple model, PyTorch is over 4x faster than TensorFlow for inference. Of course, this is just one specific example on one hardware setup. The relative performance may differ for different model architectures, batch sizes, hardware, etc. The point is that timeit makes it easy to do these kinds of comparative timings.

We can take this analysis a step further by using timeit‘s repeat function to get statistics over multiple runs:

num_repeats = 10
pytorch_times = timeit.repeat(stmt=pytorch_stmt, setup=pytorch_setup, number=num_iters, repeat=num_repeats)
tensorflow_times = timeit.repeat(stmt=tensorflow_stmt, setup=tensorflow_setup, number=num_iters, repeat=num_repeats)

print(f"PyTorch times: {pytorch_times}")  
print(f"PyTorch mean: {np.mean(pytorch_times):.5f} seconds")
print(f"PyTorch min: {np.min(pytorch_times):.5f} seconds")
print(f"PyTorch max: {np.max(pytorch_times):.5f} seconds")

print(f"TensorFlow times: {tensorflow_times}")
print(f"TensorFlow mean: {np.mean(tensorflow_times):.5f} seconds") 
print(f"TensorFlow min: {np.min(tensorflow_times):.5f} seconds")
print(f"TensorFlow max: {np.max(tensorflow_times):.5f} seconds")

This repeats the timing 10 times for each framework and prints out the raw times along with the mean, min, and max. On my machine, this outputs:

PyTorch times: [0.3405020430000572, 0.33365580100022555, 0.3332727909999938, 0.3322594170000729, 0.3329733830000031, 0.33386579699997157, 0.3321638600000739, 0.33263159800000095, 0.33264285000002495, 0.3321609020000063]
PyTorch mean: 0.33304 seconds
PyTorch min: 0.33216 seconds 
PyTorch max: 0.34050 seconds

TensorFlow times: [1.5342138260000436, 1.539452859999955, 1.5629601039999723, 1.5664960579999922, 1.528362580000007, 1.533841030999977, 1.534588986999991, 1.536534287000031, 1.5326102440000216, 1.5290506909999997]  
TensorFlow mean: 1.54312 seconds
TensorFlow min: 1.52905 seconds
TensorFlow max: 1.56650 seconds

We can see there‘s a bit of variability between runs, but PyTorch is consistently faster for this model. Doing multiple repeats like this and looking at the statistical distribution of times can give you higher confidence in your timings.

Best Practices for Timing AI/ML Code

When using timeit to measure AI and ML code, there are a few best practices to keep in mind:

1. Use a sufficiently large number of iterations. Many ML operations, especially on the GPU, have a high degree of variability in their execution time due to factors like dynamic clock speeds, caching, and memory transfers. Using a large number of iterations (thousands or more) helps average out this variability and get a more stable measurement.

2. Clear GPU memory between measurements. If you‘re timing GPU code, it‘s important to clear the GPU memory between timing runs to ensure a fair comparison. In PyTorch, you can use torch.cuda.empty_cache(). In TensorFlow, use tf.keras.backend.clear_session().

3. Disable gradient computation for inference timings. When timing inference code, you can often get a speedup by disabling gradient computation, since it‘s not needed for inference. In PyTorch, wrap the model‘s forward pass with torch.no_grad(). In TensorFlow, use tf.keras.backend.set_learning_phase(0) to disable training mode.

4. Use realistic input data. For your timings to be representative of real-world performance, use input data with shapes and value distributions similar to what you expect in production. Synthetic random data is OK for comparative timings, but real data is better for absolute performance measurements.

5. Time on target hardware. AI/ML performance is highly dependent on hardware, especially the GPU. Always do your timings on the same type of hardware you plan to use in production for the most relevant results.

Alternative Performance Tools

While timeit is great for quickly comparing code snippets, for more detailed performance analysis you may want to use a full-fledged profiler. Python includes a built-in profiler called cProfile that can provide a function-level breakdown of where your code is spending its time. There are also a number of third-party Python profilers like py-spy and pyinstrument that provide additional features.

For GPU code, the major deep learning frameworks provide profiling tools specifically for measuring and visualizing the performance of model training and inference. For example, PyTorch has the torch.utils.bottleneck profiler, and TensorFlow has the TensorBoard profiler. These tools can provide detailed insights into GPU utilization, kernel execution times, memory transfers, and more.

Hardware Considerations

As mentioned earlier, AI/ML performance is highly dependent on hardware. When doing performance measurements, it‘s critical to use hardware that‘s representative of your production environment. In particular, the choice of GPU can have an enormous impact on training and inference times.

If you‘re doing comparative measurements (e.g. framework A vs. framework B), it‘s important to do the comparisons on the same hardware to ensure a level playing field. However, even with identical hardware, other factors like driver versions, CUDA versions, and framework builds can impact performance. For fully reproducible results, you may want to use standardized Docker images for your timing environment.

When it comes to choosing hardware for AI/ML, there are a lot of factors to consider: GPU architecture (NVIDIA vs. AMD), GPU memory size and bandwidth, CPU, RAM, storage, etc. Cloud vs. on-prem is another major decision. Doing your own timings with representative workloads is the best way to make informed hardware choices for your specific use case.

Conclusion

Measuring and optimizing performance is a key part of AI/ML development at every stage, from research to production. The Python timeit library provides a simple but powerful way to compare the execution speed of different code snippets and implementations. By following best practices like using a large number of iterations, clearing GPU memory between runs, and using realistic input data, you can get performance measurements that translate into real-world efficiency gains.

Beyond timeit, profiling tools can give you more detailed insights into the performance bottlenecks in your code, and using the right hardware for your workload is critical for getting optimal performance.

Ultimately, faster AI/ML code means faster progress – whether that‘s quicker iterations in the research lab or lower latency and cheaper inference in production. Any time you invest in making your code more efficient is likely to pay off many times over. So get out there and start optimizing – your models (and your users) will thank you!