# A Deep Dive into Self\-Supervised and Supervised Contrastive Learning

- Canonical: https://33rdsquare.com/a-detailed-study-of-self-supervised-contrastive-loss-and-supervised-contrastive-loss/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

In recent years, contrastive learning has emerged as a powerful technique for representation learning in a variety of domains, from computer vision to natural language processing. Contrastive learning aims to learn an embedding space where similar samples are mapped close together and dissimilar ones are pushed apart. This simple yet effective learning framework has led to huge advances, especially in self-supervised learning where useful representations can be learned from unlabeled data.

In this post, we‘ll take an in-depth look at two key paradigms of contrastive learning: self-supervised contrastive learning (SSCL) and supervised contrastive learning (SCL). We‘ll examine how they work, their similarities and differences, and their applications and impact. Let‘s dive in!

## Self-Supervised Contrastive Learning

Self-supervised learning aims to learn meaningful representations from unlabeled data by defining a pretext task that the model must solve, in lieu of explicit labels. Contrastive learning has become one of the most successful approaches to self-supervised learning, especially for visual data.

The key idea of SSCL is to learn an embedding space where different augmented views of the same image (positive pairs) are pulled together, while views from different images (negative pairs) are pushed apart. More formally, SSCL typically uses the following components:

- A data augmentation module that generates correlated views of each image (e.g. random crops, color distortion, etc.)
- A feature extraction network f(.) (e.g. a ResNet encoder) that maps each view to an embedding vector
- A small MLP projection head g(.) that transforms the embeddings to the space where contrastive loss is applied
- A contrastive loss function that enforces similarity between positive pairs and dissimilarity between negative pairs

One common choice for the contrastive loss is the InfoNCE loss, which is based on noise contrastive estimation (NCE). For a positive pair (i, j), the InfoNCE loss is defined as:

![](https://miro.medium.com/max/1400/1*K0aHoHISCcN48uTryPZQfA.png)

Here τ is a temperature parameter, (i,j) is a positive pair, and Ni is the set of negative samples for i. fi=f(xi) is the embedding of view xi, and similarly for fj. The InfoNCE loss is the log loss of classifying the positive sample correctly among a set of distractors.

By minimizing the contrastive loss over a large number of both positive and negative pairs, SSCL can learn embedding spaces with powerful semantic structure, capturing meaningful similarities and differences between images. These pretrained embeddings can then be leveraged for efficient transfer learning on downstream tasks like image classification, object detection, etc. with limited labeled data.

Self-supervised contrastive learning has been hugely successful across many applications. In computer vision, models like SimCLR, MoCo, BYOL, and more have shown that SSCL can learn visual representations that rival or exceed those from supervised ImageNet pretraining. In NLP, contrastive pretraining like RoBERTa has also become a core component of language model pretraining. The power of learning from unlabeled data in this way has made SSCL a vital technique in modern ML.

## Supervised Contrastive Learning

While SSCL thrives on learning from unlabeled data, the recent supervised contrastive learning paradigm asks: can we also leverage label information to improve contrastive representation learning? SCL shows that the answer is a resounding yes — explicitly incorporating label structure into the contrastive loss can significantly boost the quality of the learned embeddings.

The key insight of SCL is that we don‘t just want to contrast semantically similar and dissimilar pairs, but rather we want to explicitly pull together samples from the _same class_ while pushing apart those from _different classes._ In the supervised setting, we have access to the true labels of our data, so we can define positive pairs as images from the same class, and negative pairs as images from different classes.

More precisely, the SCL loss is defined as follows. Let {(xi,yi)} be a batch of labeled data. Then for each image xi, we define:

- Pi = {xj : yj = yi, j != i} (the set of positives for xi — other images from the same class)
- Ni = {xj : yj != yi} (the set of negatives for xi — images from different classes)

Then the supervised contrastive loss for a positive pair (xi, xp) is:

![](https://production-media.paperswithcode.com/methods/Screen_Shot_2021-02-20_at_1.52.45_PM.png)

where f(x) is the embedding of x, and τ is the temperature. The loss encourages the model to assign high probability to positive pairs (xi, xp) while minimizing the probability assigned to negative pairs (xi, xk).

A key advantage of SCL is that it provides a much richer supervisory signal than the standard cross-entropy loss. Cross-entropy only compares each sample to the class prototypes, while SCL contrasts each sample with _every other sample_ in the batch. This allows SCL to learn more discriminative features that aren‘t just separable, but also more tightly clustered within each class.

![](https://www.analyticsvidhya.com/wp-content/uploads/2023/05/Supervised-Contrastive-Learning-1024x392.png)

The figure above from the original SCL paper illustrates this well — the SCL embeddings (right) show cleaner separation between classes and tighter clustering within classes compared to cross-entropy (left). This leads to enhanced visual representations that can significantly improve performance on downstream tasks.

For example, the authors showed that simply replacing the cross-entropy loss with SCL during ImageNet training boosted top-1 accuracy by nearly 1% for a ResNet-50 model. Improvements of 1-2% were also seen on fine-grained image classification datasets like FGVC Aircraft and Stanford Cars.

Here‘s a PyTorch code snippet implementing the key parts of the SCL loss:

```
def supervised_contrastive_loss(features, labels, temperature=0.07):
  """Supervised contrastive loss from https://arxiv.org/pdf/2004.11362.pdf"""

  batch_size = features.shape[0]
  labels = labels.contiguous().view(-1, 1)
  mask = torch.eq(labels, labels.T).float()

  contrast_feature = features
  anchor_feature = contrast_feature
  anchor_dot_contrast = torch.div(
    torch.matmul(anchor_feature, contrast_feature.T),
    temperature
  )

  # Normalize logits across negatives for numerical stability
  logits_max, _ = torch.max(anchor_dot_contrast, dim=1, keepdim=True)
  logits = anchor_dot_contrast - logits_max.detach()

  # Mask out self-comparisons
  logits_mask = torch.scatter(
    torch.ones_like(mask),
    1,
    torch.arange(batch_size).view(-1, 1),
    0
  )
  mask = mask * logits_mask

  # Compute log-probabilities
  exp_logits = torch.exp(logits) * logits_mask
  log_prob = logits - torch.log(exp_logits.sum(1, keepdim=True))

  # Compute mean of log-likelihood over positives
  mean_log_prob_pos = (mask * log_prob).sum(1) / mask.sum(1)
  loss = -mean_log_prob_pos.mean()

  return loss
```

The key steps are:

1. Compute the pairwise cosine similarities between all samples in the batch (anchor_dot_contrast)
2. Normalize the logits by subtracting the max for numerical stability
3. Mask out self-comparisons
4. Compute the log-probabilities for each positive and negative pair
5. Take the mean of the log-likelihoods over all positive pairs

While the expression looks complex, the core idea is intuitive — we want the model to assign high probability to true positive pairs and low probability to all other pairs.

The authors also showed how to generalize SCL to multiple positives per anchor point. If we have M positives per anchor (including the anchor itself), the mask and loss computation becomes:

```
mask = mask.repeat(anchor_count, contrast_count)
mask = mask * logits_mask

mean_log_prob_pos = (mask * log_prob).sum(1) / (mask.sum(1) + 1e-12)
loss = -mean_log_prob_pos.view(anchor_count, batch_size).mean()
```

Increasing the number of positives acts as a stronger regularizer and can lead to even better representations. The authors found that using M=2 positives worked best, outperforming both M=1 (single positive) and higher values.

To summarize, supervised contrastive learning leverages label information to improve the structure of the learned embeddings, pulling together samples within each class while pushing apart different classes. This leads to significant improvements on supervised transfer learning tasks compared to the standard cross-entropy loss.

## Self-Supervised vs. Supervised Contrastive Learning

Having examined both self-supervised and supervised contrastive learning in detail, let‘s briefly compare and contrast the two paradigms:

Similarities:

- Both aim to learn embedding spaces where similar items are close and dissimilar items are far apart
- Both utilize a contrastive loss based on positive and negative sample pairs
- Both can be used for unsupervised pretraining to learn transferable representations

Differences:

- SSCL operates on unlabeled data and learns similarities between augmented views of the same image, while SCL has access to label information and contrasts samples based on class membership
- SSCL is often used to pretrain representations that are then transferred to downstream supervised tasks, while SCL is typically used to directly train models for supervised tasks
- SCL can take advantage of multiple positives per anchor point, while SSCL usually uses just one positive (augmented) view per image

Despite these differences, the two approaches are highly complementary. Self-supervised contrastive pretraining has become a standard technique to initialize models for supervised learning, while supervised contrastive loss can further enhance representation learning during supervised training itself. An exciting direction for future work is to combine the strengths of both into a unified framework for semi-supervised learning.

## Conclusion and Future Directions

Contrastive learning has proven to be an immensely powerful tool for representation learning, both with and without labels. Self-supervised contrastive learning can discover rich, transferable visual features from unlabeled data, while supervised contrastive learning can boost fully-supervised models by learning more separable and compact within-class embeddings.

As we‘ve seen, the key ingredients are:

1. Effective data augmentation or generation of contrasting positive and negative pairs
2. A contrastive loss that attracts similar pairs and repels dissimilar pairs
3. A scalable architecture for computing all pair-wise similarities within a batch

With these components, contrastive learning can be extended and adapted to a wide range of settings. Recent techniques like BYOL and SimSiam show that the negative pairs can even be omitted entirely, learning from only positive views of each image. Other works explore combining contrastive learning with generative models, equivariant architectures, and more.

Exciting future directions include:

- Scaling contrastive learning to ever larger unlabeled datasets and model sizes
- Unifying contrastive learning with other self- and semi-supervised paradigms
- Applying contrastive learning to modalities beyond images, such as video, speech, and multimodal data
- Exploring the theoretical foundations of contrastive learning and its connections to mutual information estimation and other frameworks

As the scale of unlabeled data continues to grow, self-supervised learning will only become more important. Contrastive learning is a key piece of the puzzle, offering a principled, flexible, and empirically successful approach to learning rich and transferable representations. We‘ve only scratched the surface of what‘s possible, and many fruitful research directions remain wide open.

We hope this post has given you a solid foundation in the key concepts and techniques behind contrastive learning. For further reading, check out the references below, and happy contrasting!

## References

[1] Chen, T., Kornblith, S., Norouzi, M. and Hinton, G., 2020. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709.

[2] He, K., Fan, H., Wu, Y., Xie, S. and Girshick, R., 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9729-9738).

[3] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, C., Pires, B.A., Guo, Z.D., Azar, M.G. and Piot, B., 2020. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733.

[4] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C. and Krishnan, D., 2020. Supervised contrastive learning. arXiv preprint arXiv:2004.11362.

[5] Chen, X. and He, K., 2020. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566.

---

Source: [A Deep Dive into Self\-Supervised and Supervised Contrastive Learning](https://33rdsquare.com/a-detailed-study-of-self-supervised-contrastive-loss-and-supervised-contrastive-loss/)