Tutorial on MNIST Digit Classification Using ClearML

Introduction to ClearML

ClearML is an open source MLOps platform that helps data scientists and machine learning engineers streamline their model development workflow. It provides a suite of tools for experiment tracking, model management, workflow orchestration, and more, allowing you to focus on building great models while it handles the operational overhead.

Some of the key features and benefits of ClearML include:

• Automatic logging of code, data, models, hyperparameters, and results for every experiment
• Easily compare and visualize experiments in a centralized UI to track progress and gain insights
• Version and manage datasets and models with data and model stores
• Scale and automate ML workflows with orchestration and pipeline capabilities
• Reproduce and collaborate on experiments with automatic environment tracking
• Integrate with your favorite tools like Jupyter, TensorFlow, PyTorch, scikit-learn, etc.

So in summary, ClearML is an end-to-end solution for implementing and automating MLOps best practices in your workflow and code base.

In this tutorial, we‘ll walk through how to use ClearML to build an MNIST handwritten digit classification model with PyTorch and Jupyter notebooks. We‘ll go through each step of loading data, building and training a model, logging results, and analyzing them in ClearML. Let‘s get started!

Tutorial Overview

Here‘s an overview of what we‘ll cover in this tutorial:

1. Installing ClearML and setting up your environment
2. Initializing a ClearML Task and configuring it in code
3. Loading and preprocessing the MNIST dataset
4. Building and training a PyTorch neural network model
5. Logging hyperparameters, metrics, model checkpoints, and more to ClearML
6. Comparing experiment results in the ClearML web UI

By the end, you‘ll have a working digit classification model and experiment log captured in ClearML. Let‘s go through each step.

Environment Setup

First, make sure you have Python 3 and pip installed. We recommend creating a virtual environment to isolate the dependencies for this tutorial.

Install ClearML with pip:

pip install clearml

ClearML is available on PyPI and can be installed like any other Python package. It comes with PyTorch and TensorFlow support out of the box.

Next, sign up for a free account at https://app.clear.ml/ to get access to the ClearML server.

Create credentials for your account under Settings > Workspace:

Add those credentials to your environment:

import os
os.environ["CLEARML_API_ACCESS_KEY"] = "access-key-here"
os.environ["CLEARML_API_SECRET_KEY"] = "secret-key-here" 
os.environ["CLEARML_API_HOST"] = "https://api.clear.ml"

from clearml import Task

These credentials will allow the ClearML Python SDK to communicate with the ClearML server to log your experiments.

Initializing a ClearML Task

With the setup out of the way, let‘s start tracking our first experiment!

In your training script or notebook, initialize a ClearML Task:

task = Task.init(project_name="mnist",
                 task_name="mnist_classification"
                 output_uri=True)  

This creates a new Task (experiment) in the "mnist" project. ClearML will automatically create the project if it doesn‘t exist. The output_uri argument tells ClearML to capture the console output.

We can log arbitrary key-value hyperparameters to the Task:

task.connect({"arch": "cnn",
              "dataset": "mnist",
              "loss": "cross entropy",  
              "optimizer": "adam",
              "lr": 0.001})

ClearML will associate these hyperparameters with the experiment and make them available for querying and analysis.

Loading the Dataset

Now let‘s load the MNIST handwritten digit dataset:

from torchvision.datasets import MNIST

train_data = MNIST("data",  
                   train=True,
                   download=True,
                   transform=transforms.ToTensor())

test_data = MNIST("data",
                  train=False, 
                  transform=transforms.ToTensor())

This downloads the MNIST dataset and applies some basic preprocessing like converting images to PyTorch Tensors. We split it into train and test sets.

Let‘s log a sample of the data with the ClearML Logger:

from clearml.binding.frameworks.pytorch import PytorchDatasetMismatchError

logger = task.get_logger()

image, label = train_data[0]
logger.report_image("sample", "image sample", iteration=0, image=image.permute(2,1,0))  

This logs the first image from the training set to ClearML, which we can view in the web UI under the experiment‘s debug samples.

Building the Model

Next let‘s define a simple convolutional neural network in PyTorch for classifying the digits:

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2(x), 2)) 
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

model = Model()

It consists of two convolutional layers followed by two fully-connected layers. ReLU activations and max pooling are applied after each conv layer.

Let‘s visualize this model architecture with ClearML:

logger.report_graph(
    series="model",
    graph_string=str(model),
    iteration=0)

ClearML automatically captures the model graph definition and visualizes it in the UI. This is useful for understanding complex models.

Training the Model

Now we‘re ready to train the model. Let‘s define the training loop:

train_loader = DataLoader(train_data, batch_size=64, shuffle=True)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

for epoch in range(5):
    for batch_idx, (data, target) in enumerate(train_loader):

        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)

        loss.backward()
        optimizer.step()

        if batch_idx % 100 == 0:
            print(f"epoch: {epoch} loss: {loss.item()}")
            logger.report_scalar(series=‘loss‘,
                                 value=loss.item(), 
                                 iteration=epoch * len(train_loader) + batch_idx)

For each epoch, we iterate through the training data in batches, make predictions with the model, calculate the loss, and optimize the parameters with gradient descent.

We use ClearML‘s Logger to log the training loss at every 100 batches, associating it with the current iteration number.

After training, let‘s evaluate the model‘s accuracy on the test set:

test_loader = DataLoader(test_data, batch_size=64, shuffle=False)

model.eval()
correct = 0
with torch.no_grad():
    for data, target in test_loader:
        output = model(data)
        pred = output.argmax(dim=1, keepdim=True)
        correct += pred.eq(target.view_as(pred)).sum().item()

accuracy = 100 * correct / len(test_data)
print(f"Test accuracy: {accuracy}%")

logger.report_metric(key="test_accuracy", 
                     value=accuracy,
                     iteration=epoch)

We switch the model to eval mode, make predictions on the test batches, and calculate the total accuracy. Finally, we log the test accuracy to ClearML.

Viewing Results in the ClearML UI

With the model trained, let‘s see how to analyze the results in the ClearML web UI.

Navigate to the Projects page and click on the "mnist" project:

This brings up the list of experiments in the project. Click on the "mnist classification" experiment:

Here we get an overview of the experiment, including the hyperparameters, model graph, and output console log.

The "SCALARS" tab shows plots of the metrics we logged, like training loss and test accuracy:

We can see how the model‘s performance improved over the training process.

For a closer look, we can compare multiple experiments side-by-side to see the effect of different hyperparameters:

ClearML makes it easy to visualize and compare different runs to gain insights for model tuning.

Benefits of ClearML for Jupyter Notebooks

While we used a standard Python script in this tutorial, ClearML is easily integrated with Jupyter notebooks for interactive model development. Some of the key benefits include:

• Automatic code and notebook versioning
• Tracking notebook state and output
• Remote access and editing of running notebooks
• Access to ClearML functions and UI within the notebook

ClearML‘s Jupyter integration provides a seamless way to scale notebooks to production pipelines and datasets.

Next Steps

Congratulations on completing this introductory tutorial on MNIST classification with ClearML! You now have the basic workflow for tracking experiments, managing models, and analyzing results.

Here are some suggestions for next steps:

• Experiment with different model architectures and hyperparameters to improve performance
• Use ClearML‘s hyperparameter optimization features to automate model tuning
• Explore remote execution options to run experiments in the cloud or on-premise clusters
• Integrate other frameworks like TensorFlow or scikit-learn into your workflow
• Set up model serving pipelines for deploying your trained models

The ClearML documentation provides many more examples and guides for implementing MLOps best practices in your projects.

Conclusion

MNIST digit classification is a classic machine learning problem that demonstrates the key parts of a model development workflow. Using ClearML, we can efficiently build and iterate on models while automatically tracking experiments and results.

The ClearML platform streamlines many of the operational challenges of large-scale machine learning, allowing data scientists to focus on training high quality models. By capturing the code, data, and outputs of each experiment, it enables reproducibility, collaboration, and automation.

Jupyter notebooks provide an interactive interface for model development, which ClearML can augment with versioning, artifact tracking, and scalable orchestration.

Whether building simple models like MNIST or productionizing deep learning, ClearML helps realize the potential of MLOps for driving impactful machine learning projects.