Demystifying the Mathematics of Convolutional Neural Networks

Convolutional neural networks (CNNs) have revolutionized the fields of computer vision and machine learning over the past decade. With their remarkable ability to automatically learn rich, hierarchical representations from raw image data, CNNs now power a wide range of technologies, from facial recognition systems and autonomous vehicles to medical image analysis and creative tools like deep fake generators.

But what‘s really going on beneath the hood of these powerful models? How do a bunch of simple mathematical operations, stacked together in a deep learning architecture, achieve such incredible results? In this post, we‘ll peel back the layers and dive into the mathematics that bring convolutional neural networks to life.

The Building Blocks of CNNs

At their core, CNNs are composed of three main types of layers stacked on top of each other in various arrangements:

1. Convolutional layers
2. Pooling layers
3. Fully connected layers

The secret sauce of CNNs lies in the convolutional layers, which learn visual features and build increasingly sophisticated representations of the input images as we move deeper into the network. Let‘s start by taking a closer look at the mathematics of these fundamental building blocks.

Convolutional Layers: The Workhorse of CNNs

The convolutional layer is where the magic happens in a CNN. Its job is to slide filters, also known as kernels or feature detectors, across an input feature map and output a new feature map indicating the presence of learned patterns at each location.

The Convolution Operation

At the heart of the convolutional layer is the convolution operation, which is simply an element-wise product between a filter and a patch of the input, followed by a sum. In the typical 2D convolution used in CNNs, a filter F is convolved with an input patch X to produce a single output value:

$$ out(i,j) = \sum_m \sum_n X(i+m, j+n) \cdot F(m, n) $$

Here, X is the input patch centered at location (i,j), F is the filter, and · denotes element-wise multiplication. By repeating this operation at each location, we get a complete output feature map.

The output size of a conv layer is determined by the input size (W), filter size (F), stride (S), and padding (P) according to:

$$ \text{output size} = \frac{W – F + 2P}{S} + 1 $$

where the output size, input size, and filter size are measured in either the height or width dimension. Stride refers to the number of pixels the filter shifts between each application, while padding involves expanding the input by a certain number of pixels on each side, usually filled with zeros.

Sparse Interactions and Parameter Sharing

Two key properties of conv layers are sparse interactions and parameter sharing. Sparse interactions means that each output value depends only on a small local region of the input, rather than the entire input as in fully connected layers. This enables CNN features to tolerate translation, scale, and distortion of objects in the image.

Parameter sharing refers to the fact that the same filter weights are re-used at every position, rather than learning a separate weight for every input location. This greatly reduces the number of learnable parameters compared to fully connected layers.

Activation Functions

After the convolution, the output feature map is typically passed through a non-linear activation function to introduce non-linearities into the model. The most common activation in modern CNNs is the rectified linear unit (ReLU):

$$ \text{ReLU}(x) = \max(0, x) $$

which simply thresholds the input at zero. Other activations like sigmoid, tanh, and leaky ReLU are also sometimes used.

Pooling Layers: Downsampling Feature Maps

Between successive conv layers, it‘s common to periodically insert pooling layers to progressively downsample the spatial size of the feature maps. This helps reduce computation and build some translation invariance into the model.

The most common type of pooling is max pooling, which outputs the maximum value in each local neighborhood:

$$ \text{MaxPool}(X)(i,j) = \max_{m,n} X(i \cdot s + m, j \cdot s + n) $$

where s is the pooling stride. Average pooling, which outputs the mean value in each neighborhood, is also sometimes used:

$$ \text{AvgPool}(X)(i,j) = \frac{1}{MN} \sum_m \sum_n X(i \cdot s + m, j \cdot s + n) $$

where M and N are the height and width of the pooling neighborhood.

Fully Connected Layers

After several rounds of convolution and pooling, the feature maps are flattened and passed through one or more fully connected layers to perform classification or regression on the extracted features. Each neuron in a fully connected layer is connected to every neuron in the previous layer.

If the feature map input to the fully connected layer has shape (height, width, channels) and the layer has N neurons, then it will have (height · width · channels · N) learnable weights, plus N bias terms. The output $z$ is computed as:

$$ z = W \cdot X + b $$

where W is the weight matrix, X is the flattened input, and b is the bias vector. The output then has shape (N,), and is typically passed through a final activation function to produce class probabilities or regression values.

Learning the Parameters: Backpropagation in CNNs

Like other neural networks, CNNs are typically trained using gradient descent and backpropagation to learn the filter weights and fully connected layer parameters that minimize a loss function. During backprop, the gradients are computed layer by layer from the output back to the input.

For a conv layer with filters F and input X, the gradient of the loss L with respect to the filters is:

$$ \frac{\partial L}{\partial F} = \sum_i \sum_j \frac{\partial L}{\partial out(i,j)} \cdot X(i,j) $$

This is essentially just the convolution of the upstream gradient $\frac{\partial L}{\partial out}$ with the layer input X, summed over all positions (i,j).

The gradient with respect to the input of the conv layer, which is needed to backpropagate to deeper layers, is a bit trickier:

$$ \frac{\partial L}{\partial X} = \text{Conv}(\frac{\partial L}{\partial out}, \text{Rot180}(F)) $$

Here $\text{Conv}$ is the convolution operator and $\text{Rot180}$ rotates the filter weights by 180 degrees.

In a pooling layer, the gradients are backpropagated only to the inputs that were selected by the pooling operation (e.g. the maximum values in each neighborhood for max pooling).

For the fully connected layers, the gradients with respect to the weights W and biases b are computed using the standard backpropagation formulas:

$$ \frac{\partial L}{\partial W} = \frac{\partial L}{\partial z} \cdot X^T $$

$$ \frac{\partial L}{\partial b} = \frac{\partial L}{\partial z} $$

where $\frac{\partial L}{\partial z}$ is the upstream gradient backpropagated from the layer output z.

By recursively applying these backpropagation formulas layer by layer, we can compute the gradients of the loss with respect to all learnable parameters in the model. These gradients are then used to update the parameters using an optimization algorithm like stochastic gradient descent (SGD), Adam, or RMSprop.

Modern CNN Architectures

Since the early successes of CNNs like LeNet-5 in the 1990s and AlexNet in 2012, CNN architectures have continued to evolve and push state-of-the-art performance to new heights. Here are a few key milestones:

• VGGNet (2014) demonstrated the power of simply stacking more conv layers with small 3×3 filters. The largest VGG models had up to 19 layers.

• GoogLeNet (2015) introduced a more complex architecture with "inception" modules that performed multiple parallel convolutions with different sized filters and concatenated the results. It also added auxiliary classifiers to intermediate layers to combat the vanishing gradient problem.

• ResNet (2016) made it possible to train extremely deep networks, over 100 layers, by introducing residual skip connections that allow gradients to flow around layers, alleviating vanishing gradients.

More specialized architectures have also been developed for tasks beyond image classification, like object detection (R-CNN, YOLO, SSD), semantic segmentation (U-Net, DeepLab), and generative modeling (DCGAN, VAE, VQ-VAE).

Practical Considerations for Training CNNs

There are a number of practical considerations and techniques that go into successfully training state-of-the-art CNNs:

• Regularization: Deep CNNs have a huge number of parameters and can easily overfit small datasets. Common regularization techniques include L2 weight decay, dropout, and data augmentation (e.g. random cropping, flipping, color jittering).

• Transfer Learning: It‘s often effective to start with a CNN pre-trained on a large dataset like ImageNet, and fine-tune it on a target dataset, rather than training from scratch. The pre-trained model provides a good weight initialization that captures general image features.

• Hyperparameter Tuning: The performance of CNNs can be quite sensitive to hyperparameters like the learning rate, momentum, batch size, weight initialization, etc. Careful tuning, guided by validation performance, is important to get the best results.

• Hardware and Software: Training large CNNs requires significant computational resources, often from multiple GPUs or TPUs. Efficient CNN primitives implemented in deep learning frameworks like TensorFlow, PyTorch, Caffe, and CNTK also play a key role.

Applications of CNNs

Since their breakout success at the ImageNet competition, CNNs have become the dominant approach for virtually all computer vision tasks, replacing painstakingly engineered visual features. Some exciting applications include:

• Image Classification: Labeling images according to a set of categories, as in the original ImageNet task. CNNs are now used to classify everything from galaxies to skin lesions.

• Object Detection: Identifying and localizing objects with bounding boxes in an image. Popular approaches include region proposal methods (R-CNN) and single-shot detectors (YOLO, SSD).

• Semantic Segmentation: Assigning a category label to each pixel in an image to delineate regions like road, sky, pedestrians, etc. Fully convolutional networks like U-Net and DeepLab are the current state-of-the-art.

• Image Generation: Synthesizing plausible new images, often conditioned on an input image or label. Generative adversarial networks (GANs) and variational autoencoders (VAEs) powered by CNNs have enabled applications like style transfer, super-resolution, image inpainting, and the infamous deep fakes.

Conclusion

Well, that was a whirlwind tour through the mathematics powering the deep learning revolution in computer vision. We‘ve seen how the elegantly simple convolution operation, coupled with some pooling, non-linearities, and backpropagation, can learn to extract astonishingly rich hierarchical representations from raw pixels.

Of course, CNNs are not without limitations — they can be data-hungry, brittle to adversarial perturbations, and biased in inscrutable ways — but there‘s no doubt they‘ve had a profound impact. As CNN architectures and training methods continue to evolve alongside growing computational resources, it‘s exciting to imagine what new frontiers in intelligent visual perception will be crossed in the coming years.