Load high-res images

ByJordan Brown

Introduction

As data scientists and machine learning practitioners, we often work with image and video data for various computer vision tasks. One common challenge is dealing with low-resolution or degraded images. While traditional upsampling methods like bicubic interpolation can help, deep learning techniques using convolutional neural networks (CNNs) have emerged as a powerful way to reconstruct and enhance the resolution of images.

In particular, autoencoders provide an unsupervised learning approach for this image super-resolution task. By training an autoencoder to compress an image into a lower-dimensional latent space representation and then reconstruct it back to its original resolution, we can leverage the model to take a low-res image and produce a higher-quality output.

In this post, we‘ll dive into the fundamentals of how autoencoders work and walk through a complete example in Python of training an autoencoder on the CelebFaces dataset to enhance facial images. Whether you‘re new to autoencoders or looking to apply them to your own projects, this guide will give you a detailed look at this important deep learning architecture. Let‘s get started!

Understanding the Autoencoder Architecture

At its core, an autoencoder is a type of neural network that aims to learn an identity function to reconstruct the original input. It consists of two main components:

  1. Encoder: Compresses the input data into a lower-dimensional latent space representation. It can be viewed as a feature extraction step.

  2. Decoder: Reconstructs the original input from the latent space representation. It can be thought of as a generative model.

The key is that the latent space representation in the middle is constrained to be of much lower dimensionality than the input. This forces the autoencoder to learn a compressed representation that captures the most salient features and structures in the data. Here‘s a diagram of a basic autoencoder architecture:

Autoencoder Architecture

Common types of layers used in autoencoder neural networks include:

  • Fully-connected (dense) layers
  • Convolutional layers (for images/video)
  • Recurrent layers (for sequences)
  • Regularization layers (e.g. dropout, batch normalization)

The encoder and decoder components are typically symmetric, i.e. they use the same number and types of layers in reverse order. During training, the goal is to minimize the reconstruction loss between the final output and the original input, often using mean squared error or cross-entropy loss.

Applications of Autoencoders

Autoencoders are a powerful tool that can be applied to a variety of tasks, including:

  • Denoising: Removing noise/corruption from images, audio, etc. by training on noisy inputs and clean targets
  • Dimensionality reduction: An alternative to PCA for feature extraction and visualization
  • Anomaly detection: Identifying outliers based on high reconstruction error from the autoencoder
  • Image colorization: Converting grayscale to color images by training on grayscale inputs and color targets
  • Image super-resolution: Enhancing the resolution of a low-quality image, which we‘ll demonstrate in this post

Autoencoders have also served as a fundamental building block for more advanced generative models like variational autoencoders (VAEs) and generative adversarial networks (GANs). The key benefit is the ability to learn from unlabeled data in a self-supervised fashion.

Facial Image Super-Resolution with Autoencoders

To illustrate the power of autoencoders for image enhancement, we‘ll walk through an example of super-resolving facial images from the CelebFaces dataset. The goal is to train an autoencoder that can take a low-resolution 64×64 input image and produce a higher-quality 128×128 output.

We‘ll use Python and the Keras deep learning library to build and train our model. Here are the key steps:

1. Load and Preprocess Data

First, we need to load our facial image data and prepare it for training. We‘ll use the PIL library to read the images and NumPy to convert to floating point tensors. It‘s important to normalize pixel values to the range [0, 1].

from PIL import Image
import numpy as np
import os


hr_images = []
for filename in os.listdir(‘celeb_faces/high_res/‘):
img = Image.open(‘celeb_faces/high_res/‘ + filename)
img = img.resize((128, 128))
img = np.array(img) / 255.0
hr_images.append(img)


hr_images = np.array(hr_images)



lr_images = []
for filename in os.listdir(‘celeb_faces/low_res/‘):
img = Image.open(‘celeb_faces/low_res/‘ + filename)
img = img.resize((64, 64))

img = np.array(img) / 255.0
lr_images.append(img)


lr_images = np.array(lr_images)

We‘ll also split our data into training and validation sets:

from sklearn.model_selection import train_test_split

train_lr, val_lr, train_hr, val_hr = train_test_split(lr_images, hr_images, 
test_size=0.2, random_state=42)

2. Define the Autoencoder Model Architecture

Next, we‘ll use the Keras functional API to define the architecture of our autoencoder model. We‘ll use a series of Conv2D layers in the encoder to learn the compressed representation, followed by a bottleneck layer. The decoder will consist of Conv2DTranspose layers to upsample back to the original 128×128 resolution.

from tensorflow.keras.layers import Input, Conv2D, BatchNormalization, LeakyReLU, Conv2DTranspose
from tensorflow.keras.models import Model

def build_autoencoder(img_shape=(64, 64, 3)):
input_img = Input(shape=img_shape)


# Encoder
x = Conv2D(32, (3, 3), padding=‘same‘)(input_img)
x = BatchNormalization()(x)
x = LeakyReLU(alpha=0.2)(x)
x = Conv2D(64, (3, 3), strides=(2, 2), padding=‘same‘)(x)
x = BatchNormalization()(x)
x = LeakyReLU(alpha=0.2)(x)
x = Conv2D(128, (3, 3), strides=(2, 2), padding=‘same‘)(x)
x = BatchNormalization()(x)
x = LeakyReLU(alpha=0.2)(x)

# Decoder 
x = Conv2DTranspose(128, (3, 3), strides=(2, 2), padding=‘same‘)(x)
x = BatchNormalization()(x)  
x = LeakyReLU(alpha=0.2)(x)
x = Conv2DTranspose(64, (3, 3), strides=(2, 2), padding=‘same‘)(x)
x = BatchNormalization()(x)
x = LeakyReLU(alpha=0.2)(x)
x = Conv2D(3, (3, 3), padding=‘same‘, activation=‘sigmoid‘)(x)

model = Model(input_img, x)
return model


autoencoder = build_autoencoder()

autoencoder.summary()


This model compresses the 64x64x3 input image down to a 16x16x128 latent space representation before reconstructing it back to the target 128×128 resolution.


3. Train the Model


With our model architecture defined, we can compile it with the Adam optimizer and mean squared error loss. We‘ll then train it for 100 epochs on our low-res/high-res image pairs.


from tensorflow.keras.optimizers import Adam

autoencoder.compile(optimizer=Adam(lr=1e-4), loss=‘mse‘)


history = autoencoder.fit(train_lr, train_hr,
validation_data=(val_lr, val_hr),
batch_size=16, 
epochs=100,
verbose=2)




Training this model on a GPU is highly recommended for faster convergence. We can visualize the training progress to ensure the loss is decreasing over time:


import matplotlib.pyplot as plt

plt.plot(history.history[‘loss‘], label=‘train‘)
plt.plot(history.history[‘val_loss‘], label=‘val‘)
plt.ylabel(‘MSE Loss‘)
plt.xlabel(‘Epoch‘)
plt.legend()
plt.show()




4. Evaluate Results


Finally, we can use our trained model to super-resolve images and compare the output to the original low-res and high-res versions. We‘ll visualize results on a few example images from the validation set.


from skimage.metrics import peak_signal_noise_ratio as psnr
from skimage.metrics import structural_similarity as ssim

def evaluate_model(model, images, titles):
fig, axs = plt.subplots(1, len(images), figsize=(20, 5))
psnrs, ssims = [], []

for i, img in enumerate(images):
    if img.shape[0] != 128:
        img = model.predict(img.reshape(1, 64, 64, 3))[0] 
    else:
        psnrs.append(psnr(img, images[i-1]))
        ssims.append(ssim(img, images[i-1], multichannel=True))

    axs[i].imshow(img)
    axs[i].set_title(titles[i])
    axs[i].axis(‘off‘)

print(f"PSNR: {np.mean(psnrs):.2f}, SSIM: {np.mean(ssims):.2f}")
plt.tight_layout()
plt.show()


evaluate_model(autoencoder,

[val_lr[0], autoencoder.predict(val_lr[0].reshape(1, 64, 64, 3))[0], val_hr[0]],

[‘Low Res‘, ‘Super Res‘, ‘High Res‘])  


This will display the low-res input, super-resolved output, and original high-res image. We can also compute quantitative metrics like PSNR and SSIM to assess the perceptual quality of the enhanced images.


Some tips for improving results:


    

  • Experiment with different model architectures and hyperparameters
    • 

  • Try patch-based training to handle larger image sizes 
    • 

  • Incorporate perceptual and adversarial losses in addition to MSE
    • 

  • Use progressive resizing to scale up to higher resolutions
    • 

  • Train on a larger and more diverse dataset
    • 



Conclusion


In this post, we explored how to use autoencoders to enhance the resolution of facial images. We covered the key components of the autoencoder architecture, walked through the implementation in Python and Keras, and demonstrated the results on the CelebFaces dataset.


Autoencoders provide a powerful unsupervised approach for learning compressed representations and generating high-quality reconstructions. While we focused on a super-resolution application here, the same techniques can be applied to other image enhancement tasks like denoising and colorization.


There are many opportunities to extend this work, such as improving the model architecture, incorporating perceptual losses, and scaling up to higher resolutions. You could also explore other applications like image inpainting or combine the autoencoder with downstream supervised tasks.


For further reading, I recommend checking out the original super-resolution autoencoder paper [1], as well as more recent approaches using GANs [2] and residual networks [3]. The Keras blog also has a great tutorial on building autoencoders [4].


I hope this guide has given you a practical understanding of autoencoders and how to apply them to your own image enhancement projects. Feel free to experiment with the code and share your results!


References:

[1] Dong, C., Loy, C.C., He, K. and Tang, X., 2015. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2), pp.295-307.

[2] Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z. and Shi, W., 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4681-4690).

[3] Lim, B., Son, S., Kim, H., Nah, S. and Mu Lee, K., 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 136-144).

[4] Building Autoencoders in Keras: https://blog.keras.io/building-autoencoders-in-keras.html


      
          
        How useful was this post?      
              
        Click on a star to rate it!      
        
      
                      
  •                       
    •                   
  •                       
    •                   
  •                       
    •                   
  •                       
    •                   
  •                       
    •               
    
    
        
      Average rating 0 / 5. Vote count: 0    
    
      No votes so far! Be the first to rate this post.    
    
  
          



You May Like to Read,

Similar Posts