The Architecture of Convolutional Neural Networks: Simplified and Demystified

Convolutional Neural Networks (CNNs) have revolutionized the field of computer vision and image recognition in the last decade. From self-driving cars to facial recognition to medical image diagnosis, CNNs are the key technology powering many modern AI applications. But how exactly do CNNs work under the hood? In this post, we‘ll demystify the architecture of convolutional neural networks in an easy-to-understand way.

Images as Matrices of Pixels

To understand how CNNs process images, we first need to understand how computers "see" images. To a computer, an image is simply a matrix (or grid) of numbers, where each number represents a pixel intensity value. For grayscale images, there is a single number per pixel location representing the brightness. For color images, there are three numbers per pixel – one each for the red, green, and blue (RGB) color channels.

The numbers are usually stored as 8-bit integers, with 0 representing black and 255 representing maximum intensity. So a 256×256 color image would be represented as a 256x256x3 matrix. Each spatial location in the matrix corresponds to a pixel location in the original image.

The Problem with Fully-Connected Networks

One naive approach to building an image classification model would be to take this raw pixel matrix, flatten it into a 1D vector, and feed it into a regular fully-connected (dense) neural network. So for our example 256×256 color image, we‘d have an input vector of length 2562563 = 196,608!

There are a couple major problems with this approach:

1. Curse of dimensionality – That‘s a massive number of inputs and learnable parameters required, which grows exponentially with the image size. This leads to overfitting and inability to generalize unless we have massive labeled training datasets.

2. Loss of spatial information – By flattening the 2D image matrix into a 1D vector, we destroy all information about the spatial arrangement of pixels. The network has no inherent understanding of which pixels are near each other. Since recognizing visual concepts is all about understanding the spatial hierarchies and patterns, this is far from ideal.

Convolutional Layers to the Rescue

The key innovation of CNNs is the convolutional layer. Instead of having each neuron connected to every single input pixel, convolutional layers learn a set of small filters (kernels) that are convolved with the input image. Each filter is a matrix of learnable weights that acts as a feature detector – as it slides across the input image matrix, it computes the dot product at each location, activating when the template pattern is found.

Some key properties of convolution:

• Preserves spatial arrangement. The output is another 2D "image" matrix (aka feature map), just a filtered version of the input.
• Parameter sharing. The same filter weights are reused across the entire input. So the number of learnable weights is determined by the filter size, not the input size.
• Local connectivity. Each output neuron is only influenced by a local region of the input, allowing the filters to learn location-invariant features.

Typical filter sizes are 3×3 or 5×5. The filters are convolved with a stride, which determines the spacing between convolution locations. Padding the input with zeros allows controlling the size of the output. Given input size W, filter size F, stride S, and padding P, the output size is given by:

Output Size = (W−F+2P) / S + 1

Stacking multiple filters produces an output feature map with a depth dimension. For example, using 64 filters on a 256×256 color image input with 3×3 filters, stride 1, and padding 1 ("same" padding) produces a 256x256x64 output.

Pooling Layers for Downsampling

While convolutional layers are great at learning abstract features, the output size can still be quite large, especially with many filters. This is where pooling layers come in – they progressively downsample the spatial dimensions of the feature maps while retaining the depth dimension.

The most common type is max pooling, which slides a window across the input and takes the max value in each window region. For example, 2×2 max pooling with stride 2 will reduce the spatial dimensions by a factor of 2. Pooling provides translation invariance (robustness to small shifts) and keeps the number of parameters manageable.

Stacking Layers to Build a CNN

A complete CNN architecture consists of stacking multiple convolution and pooling layers, followed by a few fully-connected (dense) layers at the end for classification:

INPUT -> [[CONV -> RELU]N -> POOL?]M -> [FC -> RELU]*K -> FC -> SOFTMAX

• N is the number of conv layers stacked before each pooling layer (often just 1-2)
• M is the number of pooling layers (downsampling steps)
• K is number of fully-connected layers at end (usually 1-3)

The convolution-pooling layers act as hierarchical feature extractors, with earlier layers learning simple low-level features (edges, textures) and later layers combining these into more complex high-level features (object parts, faces). The fully-connected layers at the end acts as a classifier, taking the high-level features and producing class scores. Softmax activation is used to output a probability distribution over the classes.

Here‘s a code snippet in Keras showing a basic CNN architecture:

model = Sequential([
    Conv2D(32, (3,3), activation=‘relu‘, input_shape=(256,256,3)),
    MaxPooling2D((2,2)),
    Conv2D(64, (3,3), activation=‘relu‘),
    MaxPooling2D((2,2)),
    Conv2D(64, (3,3), activation=‘relu‘),
    Flatten(),
    Dense(64, activation=‘relu‘),
    Dense(10, activation=‘softmax‘)
])

Innovations in CNN Architectures

Since their introduction in the 1990s, CNN architectures have continued to evolve and reach new heights in image recognition performance. Here are some of the key CNN architecture innovations in recent years:

• AlexNet (2012) – First CNN to win ImageNet competition, 5 conv layers, 3 FC layers, introduced ReLU activation
• VGGNet (2014) – 16-19 layer CNN with uniform 3×3 conv filters, showed depth is critical
• Inception (2014) – Introduced use of various conv filter sizes and concatenation within the same layer to capture multi-scale features
• ResNet (2015) – Added residual skip connections to enable training hundreds of layers while avoiding vanishing gradients
• DenseNet (2017) – Each layer connected to every other layer in feedforward fashion to encourage feature reuse
• EfficientNet (2020) – Used neural architecture search to find optimal balance of CNN depth, width, and resolution

Conclusion

Convolutional neural networks have proven incredibly effective at computer vision tasks by leveraging the spatial structure of images. The key components – convolutional layers, pooling layers, and fully-connected layers – work together to hierarchically extract visual features and classify images.

While CNNs are most commonly associated with image data, the same architecture can be extended to other spatially-structured data like time series (1D convolutions) and video (3D spatio-temporal convolutions). Mastering the basics of CNNs will serve you well across a wide range of deep learning applications.

I hope this post helped clarify the inner workings of convolutional neural networks! Let me know in the comments if you have any other questions.