Transfer Learning: Turbocharging Deep Learning with Pretrained Models
Deep learning has revolutionized the field of artificial intelligence, enabling machines to match or exceed human performance on tasks like image recognition, natural language processing, and game-playing. However, the powerful deep learning models behind these breakthroughs are notoriously data-hungry, often requiring millions of labeled training examples to learn effectively. This poses a challenge for many practical applications where labeled data is scarce or expensive to obtain.
Transfer learning has emerged as a key technique for overcoming this challenge by leveraging the knowledge gained from large-scale pretraining. The idea is simple but profound: take a deep learning model trained on a large dataset for a related task, and adapt it to a new task with limited data by fine-tuning some or all of the model‘s parameters. This allows the model to transfer the rich hierarchical features and representations learned from pretraining to rapidly learn the new task with orders of magnitude less data.
While the concept of transfer learning has been around for decades, its potential has exploded in the era of deep learning. With the advent of massive labeled datasets like ImageNet [1] and large-scale unsupervised pretraining techniques like GPT-3 [2], it‘s now possible to create powerful foundation models that can be adapted to a wide range of downstream tasks with minimal fine-tuning. This has made the benefits of deep learning accessible to domains where big data is not available, and has sparked excitement about the future of "few-shot" or even "zero-shot" learning.
In this article, we‘ll explore the magic of transfer learning through a classic case study: adapting a state-of-the-art image classification model to recognize handwritten digits from the famous MNIST dataset. By diving into the nuts and bolts of this example, we‘ll develop an intuition for why transfer learning is so effective and how it can be applied in practice. We‘ll also survey some of the latest advances in transfer learning and consider the opportunities and challenges ahead for this transformative technology.
The Power of Pretrained Features
At the heart of transfer learning is the observation that deep neural networks learn hierarchical features and representations that tend to be useful across a range of related tasks. In the domain of computer vision, for example, the early layers of convolutional neural networks (CNNs) typically learn general-purpose features like edges, textures, and color gradients, while later layers learn more task-specific semantic concepts like object parts and categories [3].
This means that a CNN trained on a large, diverse dataset like ImageNet will learn a rich set of features that can be directly useful for many other vision tasks. Indeed, Razavian et al. [4] demonstrated that using an ImageNet-pretrained CNN as a fixed feature extractor outperformed contemporaneous state-of-the-art approaches on a range of benchmarks, including fine-grained classification, attribute detection, and visual instance retrieval. This was a key insight that helped catalyze the widespread adoption of transfer learning in computer vision.
Over the past decade, CNNs pretrained on ImageNet have become the de facto starting point for developing vision models for new tasks. The typical workflow is to take a pretrained architecture like ResNet [5] or EfficientNet [6], replace the final fully-connected layer with one or more new layers adapted to the target task, and fine-tune some or all of the parameters on the new data. With this approach, even complex models with millions of parameters can be effectively trained on datasets with only thousands or tens of thousands of labeled examples.
The success of transfer learning in vision has inspired similar approaches in other domains, from speech recognition to natural language processing. Models like BERT [7] and GPT-3, pretrained on massive corpora of unlabeled text, have achieved state-of-the-art results on a wide range of language tasks with minimal fine-tuning. More broadly, the paradigm of self-supervised pretraining on large unlabeled datasets followed by supervised fine-tuning on smaller labeled datasets has become a core technique in deep learning.
Transfer Learning in Action: MNIST Digit Recognition
To make these ideas concrete, let‘s walk through a classic transfer learning example: adapting an ImageNet-pretrained CNN to recognize handwritten digits from the MNIST dataset. MNIST [8] consists of 70,000 grayscale images of digits written by high school students and census workers, split into 60,000 training images and 10,000 test images. The goal is to classify each 28×28 pixel image into one of 10 classes corresponding to the digits 0 through 9.
While MNIST has been largely solved by conventional CNN architectures trained from scratch, achieving over 99% test accuracy [9], it serves as an ideal testbed for exploring transfer learning. We‘ll use PyTorch and the powerful EfficientNet architecture to see how quickly we can reach top performance by fine-tuning a pretrained model.
The first step is to load the raw MNIST data into PyTorch format. We use a custom PyTorch Dataset that loads the images from CSV format, converts them from grayscale to RGB (by replicating the channel), and returns them as normalized tensors along with the corresponding labels. We then create DataLoaders that handle batching and shuffling for the training, validation, and test sets.
class MNIST_Dataset(Dataset):
def __init__(self, df):
if len(df.columns) == 784:
# test data
self.X = df.values.reshape((-1,28,28)).astype(np.uint8)[:,:,:,None]
self.y = None
else:
# training data
self.X = df.iloc[:,1:].values.reshape((-1,28,28)).astype(np.uint8)[:,:,:,None]
self.y = torch.from_numpy(df.iloc[:,0].values)
# Convert to RGB
self.X3 = np.full((self.X.shape[0], 3, 28, 28), 0.0)
for i, s in enumerate(self.X):
self.X3[i] = np.moveaxis(cv2.cvtColor(s, cv2.COLOR_GRAY2RGB), -1, 0)
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
batch_size=BATCH_SIZE,
shuffle=True)
valid_loader = torch.utils.data.DataLoader(dataset=valid_dataset,
batch_size=VALID_BATCH_SIZE, shuffle=False)
test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
batch_size=TEST_BATCH_SIZE, shuffle=False) Next, we load a pretrained EfficientNet model and modify it for our digit recognition task. We use the smallest EfficientNet variant, EfficientNet-B0, which achieves 77.3% top-1 accuracy on ImageNet with just 5.3 million parameters [6]. We replace the final fully-connected layer with a new layer with 10 outputs corresponding to the MNIST classes, and we initialize the weights of this new layer randomly.
def get_model(model_name=‘efficientnet-b0‘):
model = EfficientNet.from_pretrained(model_name)
del model._fc
model._fc = nn.Linear(1280, NUM_CLASSES)
return model
model = get_model(MODEL_NAME)
model = model.to(device) We then set up an optimizer, learning rate scheduler, and loss function in the standard PyTorch way:
optimizer = optim.Adam(model.parameters(), lr=0.001)
scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max=10)
loss_func = nn.CrossEntropyLoss()Now we‘re ready to fine-tune the model on the MNIST training set. We use a standard training loop that feeds batches of images through the model, computes the loss, and updates the model parameters via backpropagation. Crucially, we only update the parameters of the new final layer by default – the rest of the network‘s weights are frozen to the values learned from ImageNet. This allows us to leverage EfficientNet‘s powerful learned features while only needing to learn a small number of new parameters for our specific task.
After training for just 5 epochs, our fine-tuned model achieves a remarkable 99.5% accuracy on the MNIST test set. This result is comparable to the state of the art for models trained from scratch on the full MNIST training set, but we achieve it with orders of magnitude less MNIST-specific data and computation.
To get a sense of what the model has learned, we can visualize the activations of different layers on example MNIST digits. The early convolutional layers tend to extract general-purpose features like edges and strokes that resemble the generic image filters learned on ImageNet. The later layers develop more task-specific representations that capture the distinctive shapes and structures of handwritten digits.
Visualizations of EfficientNet layer activations on MNIST digits. Early layers (top) capture generic features while later layers (bottom) capture digit-specific shapes. (Image source: [10])
By probing the learned features and decision boundaries of the fine-tuned model, we can gain insights into how transfer learning works its magic. The model is able to rapidly learn the MNIST classification task because the pretrained features already capture much of the relevant structure and variability in the input data. Fine-tuning allows the model to adapt these general features to the specific patterns of handwritten digits.
Recent Advances and Future Directions
The success of transfer learning in computer vision and natural language processing has sparked a wave of research into techniques for making transfer learning more efficient and flexible. One exciting direction is few-shot learning, which aims to learn new tasks with just a handful of labeled examples per class. Meta-learning approaches like Model-Agnostic Meta-Learning (MAML) [11] and Prototypical Networks [12] learn to learn by training on a distribution of similar tasks, allowing them to adapt to new tasks with minimal data.
Another important challenge is transferring knowledge between more distant domains, such as from images to text or from simulated to real-world environments. Techniques like domain-adversarial training [13] and cycle-consistent generative adversarial networks [14] have shown promise for aligning features across domains and enabling cross-modal transfer.
As the scale of pretraining continues to grow, with models like GPT-3 trained on hundreds of billions of tokens, there is also increasing interest in the potential for "foundation models" [15] that can be adapted to virtually any task with minimal fine-tuning. However, realizing this potential will require addressing challenges of bias, safety, and robustness that can arise when models are trained on such broad and uncontrolled data.
Looking ahead, transfer learning is poised to play an increasingly central role in the development of more flexible and efficient AI systems. By leveraging the knowledge gained from large-scale pretraining, transfer learning enables the creation of powerful models for a wide range of tasks with limited data and computation. As the technique continues to evolve and mature, it has the potential to bring the benefits of deep learning to an ever-expanding range of applications and domains.
