Building a Food Image Classifier using Fastai

Image classification is a common task in computer vision that involves assigning a label to an input image from a predefined set of categories. This has many useful applications, such as automatically categorizing food photos on social media, identifying dishes in restaurant reviews, classifying recipe images on cooking websites, and more.

In this post, we‘ll walk through how to build an image classifier to recognize different types of food in photos using the fastai deep learning library. Fastai is a high-level library built on top of PyTorch that makes training state-of-the-art deep learning models accessible to everyone. It provides a simple, intuitive API for defining models and includes many best practices out-of-the-box.

The general steps we‘ll follow to build our food image classifier are:

1. Prepare a labeled dataset of food images
2. Define a convolutional neural network (CNN) model architecture
3. Train the model on the dataset
4. Evaluate performance and make predictions on new images

Let‘s get started! The full code is available in this [Jupyter notebook](link to your GitHub repo).

Preparing the Data

The first step is to obtain a dataset of labeled food images to train our model on. For this example, we‘ll use the Food-101 dataset which contains 101,000 images of 101 different food categories, with 1000 images per category.

We can easily download this dataset using fastai‘s built-in untar_data function:

path = untar_data(URLs.FOOD)

This downloads the tar file, extracts it, and returns a Path object with the extracted location. The URLs collection provides quick access to several academic datasets used for benchmarking models.

Next we need to tell fastai how our data is structured. The Food-101 dataset follows a common pattern where the training and validation images are split into different folders, with subfolders for each category:

food-101/
    images/
        train/
            apple_pie/
                134.jpg
                253.jpg
                ...
            waffles/
                83.jpg
                176.jpg
                ...
            ...
        valid/
            apple_pie/
                17.jpg
                63.jpg
                ...
            waffles/ 
                82.jpg
                186.jpg
                ...
            ...

With this structure, we can create an ImageDataLoaders object by passing the path and indicating that the validation set is a 20% random split:

dls = ImageDataLoaders.from_folder(
    path, valid_pct=0.2, seed=42,
    item_tfms=Resize(224))

The item_tfms argument resizes each image to 224×224 pixels which is a standard input size for many CNN architectures. There are many other augmentations we could apply to expand the dataset, like random flips, rotations, etc.

We can visualize a batch of the data:

dls.train.show_batch(max_n=9, figsize=(7,6))

Looking good! With our data prepared, let‘s define the model.

Defining the Model

There are many possible CNN architectures we could use for image classification. However, we don‘t have to start from scratch – fastai provides easy access to many pre-trained state-of-the-art models.

Using a pre-trained model is beneficial because it has already learned to recognize many general image features on a large dataset (usually ImageNet). We can leverage this knowledge and adapt it to our specific dataset, which is called transfer learning. This greatly reduces the time and data needed to train a high accuracy model.

For this example, we‘ll use the popular ResNet-50 architecture:

learn = vision_learner(dls, resnet50, metrics=error_rate)

The vision_learner function is a convenient way to create a Learner object with an appropriate model. We pass it the data, architecture, and evaluation metric (error rate in this case).

We can inspect the model architecture:

learn.model

This prints a summary of the ResNet-50 CNN which has 50 convolutional layers. The final layer is a custom head that fastai adds to adapt the model for our Food101 classes.

Training the Model

Now we‘re ready to train the model on our data! Fastai makes this easy with the fine_tune method:

learn.fine_tune(epochs=5)

This unfreezes the model weights and trains for 5 epochs using good default learning rates and other hyperparameters. We‘ll see output like:

epoch     train_loss  valid_loss  error_rate  time
0         1.096201    0.731982    0.215419    02:04
1         0.796640    0.557245    0.171530    02:03
2         0.650022    0.492744    0.150965    02:04
3         0.568923    0.440079    0.137550    02:02
4         0.509037    0.409490    0.128483    02:02

We can see both the training and validation losses decreasing over time, which is a good sign the model is learning! The error rate on the validation set reaches 12.8% after 5 epochs. Not bad for just a few lines of code!

We can visualize how the model performs on the validation set with a confusion matrix:

interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix(figsize=(12,12), dpi=60)

The confusion matrix shows that the model does quite well overall, with most images falling along the diagonal (correct predictions). There are some categories it has more trouble distinguishing, like Apple Pie vs Waffles.

To further improve accuracy, we could try:

• Training for more epochs
• Using a larger/different CNN architecture
• Adding more image augmentation
• Getting more training data

However, 87% accuracy across 101 categories is already quite good for our purposes! Feel free to experiment with different settings in the Jupyter notebook.

Making Predictions

To use our trained model to classify new images, we can simply pass them to the predict method:

img = PILImage.create(‘donuts.jpg‘)
img.show()

pred_class,pred_idx,outputs = learn.predict(img)
print(f‘Prediction: {pred_class}‘)

Prediction: donuts

Our model correctly identifies the donuts in the image! We could integrate this into an application to automatically tag user-uploaded food photos, generate alt text descriptions, and more.

Conclusion

In this post, we saw how to train an image classifier using the fastai library to categorize photos of food with 87% accuracy. The key steps were:

1. Preparing a dataset of labeled food images
2. Defining a CNN model architecture (ResNet-50)
3. Training the model on the data with fine_tune
4. Evaluating the model and visualizing performance
5. Using the predict method to classify new images

While we used a food dataset as an example, the same approach can be applied to any type of image classification task. Fastai makes it quick and easy to train highly accurate models without having to write a lot of code.

There are many ways to build upon what we covered and take the model further:

• Try other model architectures like ResNet-18/34/152 or EfficientNet
• Increase image size for better performance (e.g. 299×299)
• Add more advanced augmentation like Mixup
• Use progressive resizing to train more efficiently
• Leverage the built-in Learning Rate Finder to optimize learning rate

I‘d encourage you to play with the code and experiment on your own! The full Jupyter notebook for this project is available on my GitHub repo.

References

• Fastai docs
• Food-101 dataset
• Deep Residual Learning for Image Recognition (ResNet)