Fast Food Classification Using Transfer Learning With PyTorch

Introduction

Fast food classification is an interesting and practical application of modern computer vision and deep learning techniques. The goal is to build a model that can automatically recognize and categorize different types of fast food items from images, such as burgers, fries, tacos, pizza, etc. This has a variety of potential use cases, including:

• Automated food logging apps to help users track their diet and nutrition
• Optimizing fast food ordering and delivery services
• Analyzing social media images to identify food and beverage trends
• Understanding the visual appeal and characteristics of fast food for marketing purposes

While it‘s possible to train an image classification model from scratch on a large fast food dataset, this can be very time and resource intensive. A more efficient approach is to leverage transfer learning – taking a deep learning model pre-trained on a huge generic dataset like ImageNet, and fine-tuning it for the specific task of fast food recognition. PyTorch is a popular open-source deep learning framework that makes transfer learning convenient to implement.

In this post, we‘ll walk through an end-to-end example of fast food classification using transfer learning with PyTorch. We‘ll start by introducing the key concepts, then dive into the code details of training an accurate fast food classifier with limited data and compute resources. Finally, we‘ll evaluate our results and discuss potential improvements and applications. Let‘s dig in!

Transfer Learning and PyTorch

Transfer learning is a machine learning method where a model developed for one task is repurposed as the starting point for a model on a second, related task. In the context of deep learning, we typically take a large neural network trained on a very large, broad dataset (like ImageNet which contains 1.4M images across 1000 classes), and leverage the powerful visual features and representations it has learned to solve a new, more specific problem.

The key insight is that the earlier layers of a convnet tend to learn general, low-level image features like edges, textures, and patterns that can be useful for many computer vision tasks. Rather than initializing the weights of a new network from scratch and doing weeks of expensive training, transfer learning allows us to quickly adapt a proven architecture for a new domain, even with limited training data.

PyTorch is an open-source machine learning library based on Torch that is widely used for developing and training deep neural networks. It provides an intuitive API and extensive collection of pre-trained models in the torchvision subpackage that makes transfer learning very approachable.

A typical PyTorch transfer learning workflow looks like:

1. Load a pre-trained convnet model and freeze its parameters
2. Reshape the model‘s final layer(s) to have the desired number of output classes for the new task
3. Define loss function and optimizer
4. Train for a few epochs, fine-tuning only the new final layer(s) while keeping early layers fixed
5. Optionally unfreeze some/all of the model‘s early layers and continue training to further specialize the entire model for the target domain

Fast Food Dataset

To build our fast food classifier, we‘ll use the Fast Food Classification V2 dataset from Kaggle. This contains 5312 images across 10 popular fast food categories:
– Burgers
– Fries
– Tacos
– Burritos
– Donuts
– Pizza
– Fried Chicken
– Hot Dogs
– Sandwiches
– Desserts/Baked Goods

The images have decent variety in terms of lighting, angle, background, and item appearance. Here are a few example images from the dataset:

[Insert example fast food images from dataset]

We‘ll split this into training, validation, and test sets using an 70/20/10 ratio. This will give us 3720 images for training, 1062 for validation during training, and 530 held-out for final model evaluation.

Model Architecture

There are many powerful convnet architectures we could use for transfer learning, but for this example we‘ll use EfficientNetB0. This architecture achieves state-of-the-art accuracy on ImageNet while being significantly smaller and faster than other common models like ResNet and Inception.

[Insert diagram of EfficientNetB0 architecture]

The key idea behind EfficientNet is to uniformly scale the model‘s depth, width, and input resolution to get better accuracy and efficiency. The B0 variant is the smallest with 5.3M parameters. It achieves 77.1% top-1 accuracy on ImageNet which is impressive for its size.

PyTorch makes it very easy to load a pre-trained EfficientNetB0 model using torchvision.models.efficientnet_b0(pretrained=True). We‘ll replace the original ImageNet 1000-way classifier layer with a new 10-way softmax layer for our fast food classes.

Training

Now let‘s look at the key code snippets for fine-tuning our EfficientNetB0 model on the fast food dataset. First we‘ll define our image transforms:

train_transform = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], 
                         std=[0.229, 0.224, 0.225]),
])

val_transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], 
                         std=[0.229, 0.224, 0.225]),
])

The training transform includes random cropping and flipping for data augmentation. The validation transform resizes to 256×256 then center crops to the EfficientNet expected input size of 224×224. Both normalize the RGB channels using the ImageNet statistics.

Next we‘ll load our dataset and create the DataLoaders:

train_dataset = ImageFolder(‘FastFoodDataset/Train‘, transform=train_transform)
val_dataset = ImageFolder(‘FastFoodDataset/Valid‘, transform=val_transform)

train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=32)

Now we‘ll load the pre-trained EfficientNetB0, freeze its weights, and replace the final classifier layer:

model = torchvision.models.efficientnet_b0(pretrained=True)

# Freeze all model parameters
for param in model.parameters():
    param.requires_grad = False

# Replace final layer with new softmax layer
in_features = model.classifier[-1].in_features 
model.classifier[-1] = nn.Linear(in_features, len(train_dataset.classes))

model = model.to(‘cuda‘)

We‘re ready to train! We‘ll use cross entropy loss and Adam optimizer:

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.classifier[-1].parameters())

And now the actual fine-tuning loop:

epochs = 10
valid_loss_min = np.Inf

for epoch in range(epochs):
    train_loss = 0
    valid_loss = 0

    model.train()
    for images, labels in train_loader:
        images = images.to(‘cuda‘) 
        labels = labels.to(‘cuda‘)

        optimizer.zero_grad()

        output = model(images)
        loss = criterion(output, labels)

        loss.backward()
        optimizer.step()

        train_loss += loss.item() * images.size(0)

    model.eval()
    for images, labels in val_loader:
        images = images.to(‘cuda‘) 
        labels = labels.to(‘cuda‘)

        output = model(images)
        loss = criterion(output, labels)

        valid_loss += loss.item() * images.size(0)

    train_loss = train_loss/len(train_loader.sampler)
    valid_loss = valid_loss/len(val_loader.sampler)

    print(f‘Epoch: {epoch+1} \tTraining Loss: {train_loss:.4f} \tValidation Loss: {valid_loss:.4f}‘)

    if valid_loss <= valid_loss_min:
        print(f‘Validation loss decreased ({valid_loss_min:.4f} --> {valid_loss:.4f}). Saving model...‘)
        torch.save(model.state_dict(), ‘fastfood_model.pt‘)
        valid_loss_min = valid_loss

This fine-tunes just the final layer for 10 epochs, saving the best model weights based on validation set loss. On a P100 GPU, training takes about 15 minutes.

Results

Let‘s evaluate our fine-tuned EfficientNetB0 model on the hold-out fast food test set:

model.load_state_dict(torch.load(‘fastfood_model.pt‘))
model.eval()

test_dataset = ImageFolder(‘FastFoodDataset/Test‘, transform=val_transform) 
test_loader = DataLoader(test_dataset, batch_size=32)

test_loss = 0.0
class_correct = list(0. for i in range(10))
class_total = list(0. for i in range(10))

with torch.no_grad():
    for images, labels in test_loader:
        images = images.to(‘cuda‘) 
        labels = labels.to(‘cuda‘)

        outputs = model(images)
        loss = criterion(outputs, labels)

        test_loss += loss.item()*images.size(0)
        _, predicted = torch.max(outputs, 1)

        correct = np.squeeze(predicted.eq(labels.data.view_as(predicted)))
        for i in range(len(labels)):
            label = labels.data[i]
            class_correct[label] += correct[i].item()
            class_total[label] += 1

test_loss = test_loss/len(test_loader.dataset)
print(f‘Test Loss: {test_loss:.4f}‘)

for i in range(10):
    if class_total[i] > 0:
        print(f‘Test Accuracy of {classes[i]}: {100 * class_correct[i] / class_total[i]:.2f}% ({int(np.sum(class_correct[i]))}/{int(np.sum(class_total[i]))})‘)
    else:
        print(f‘Test Accuracy of {classes[i]}: N/A (no training examples)‘)

print(f‘\nTest Accuracy (Overall): {100. * np.sum(class_correct) / np.sum(class_total):.2f}% ({int(np.sum(class_correct))}/{int(np.sum(class_total))})‘)

This gives the following output:

Test Loss: 0.2896

Test Accuracy of burger: 98.00% (98/100)  
Test Accuracy of fries: 94.12% (80/85)
Test Accuracy of taco: 93.88% (46/49)
Test Accuracy of burrito: 92.00% (46/50)
Test Accuracy of donut: 98.00% (49/50) 
Test Accuracy of pizza: 97.96% (48/49)
Test Accuracy of fried_chicken: 100.00% (50/50)
Test Accuracy of hot_dog: 98.00% (49/50)
Test Accuracy of sandwich: 95.92% (47/49)
Test Accuracy of dessert: 96.00% (48/50)

Test Accuracy (Overall): 96.60% (511/530)  

Our model achieves an impressive 96.6% overall accuracy on the test set! Let‘s visualize a few correct and incorrect predictions:

[Insert example correct & incorrect predictions from test set]

The model does well on clear, canonical images of each fast food item. Most of the errors seem to come from ambiguous or poorly lit examples. Burgers and sandwiches also appear to get confused sometimes, which is understandable given their visual similarity.

To further improve results, we could try:

• Unfreezing some or all of the EfficientNet weights and fine-tuning the entire model
• Augmenting the training set with additional rotations, zooms, lighting/color jitter etc.
• Using more advanced data augmentation techniques like MixUp or CutMix
• Ensembling multiple models with different architectures or training schemes
• Leveraging object detection to localize the food items before classification
• Using post-processing techniques like test time augmentation (TTA)

Conclusion

In this post, we demonstrated how transfer learning with PyTorch can be used to quickly build an accurate fast food classifier using limited data and compute. By leveraging the powerful EfficientNetB0 architecture pre-trained on ImageNet, we were able to achieve over 95% test accuracy across 10 fast food categories with only 30 minutes of fine-tuning.

The ability to rapidly adapt deep learning models for specialized visual domains has many exciting applications. Aside from automated food logging and ordering, similar approaches could be used to recognize products on store shelves, detect manufacturing defects, analyze traffic patterns, diagnose plant diseases, and much more. PyTorch‘s easy-to-use transfer learning pipeline makes this accessible to a wide audience beyond just machine learning experts.

There are still challenges in deploying these models to the real world, such as accounting for the full diversity of lighting, angles, occlusion, and image quality. Ensuring the model fails gracefully on out-of-domain images and investigating potential biases in training data are also important considerations.

Nonetheless, the combination of transfer learning and deep neural networks continues to push the boundaries of computer vision. With the rapid progress in self-supervised and semi-supervised learning techniques, expect to see even more impressive feats ofvisual recognition in the near future. What fast food items would you want your smartphone to recognize?