Convolutional Neural Networks: An In-Depth Look at CNN Filters

Artificial neural networks have revolutionized the field of machine learning and opened up incredible possibilities in areas like computer vision, natural language processing, and speech recognition. One of the most powerful and widely used types of neural networks is the convolutional neural network (CNN).

CNNs have achieved remarkable results on tasks like image classification, object detection, facial recognition, and more. They are designed to process data with a grid-like topology, such as images and videos, by automatically learning hierarchical feature representations.

At the core of how CNNs work is the convolution operation and the use of filters (also called kernels). Having a solid understanding of these concepts is key to appreciating the power of CNNs. In this post, we‘ll do a deep dive into convolution and filters, building up your intuition for how they enable CNNs to learn effectively from visual data.

Convolution 101

In mathematics, convolution is defined as the integral of the product of two functions after one is reversed and shifted. But for our purposes, you can think of convolution simply as the operation of multiplying two matrices (the input and the filter) elementwise and then summing up the results into a single output value.

Here‘s an animation showing how a 3×3 filter slides across a larger input image, performing the convolution operation at each location to produce the output feature map:

Animation of 2D convolution with a 3×3 filter. Source: https://github.com/vdumoulin/conv_arithmetic

Intuitively, you can think of the filter as a feature detector – each filter learns to detect a particular visual pattern or characteristic, like edges, textures, shapes, etc. The output value represents how strongly that feature is present at that location in the input.

The size of the output from convolution depends on:

• The input size (H_ixW_i)
• The filter size (H_fxW_f)
• The stride (S)
• The amount of zero padding (P)

The output height and width are given by:

H_o= (H_i+2P−H_f)/S + 1

W_o= (W_i+2P−W_f)/S + 1

By convention, filters are usually square with an odd size like 3×3, 5×5, 7×7 etc. This allows for a clear center pixel. The filter size is a hyperparameter that is chosen based on the input size, the complexity of features to detect, and empirical results.

The stride refers to how many pixels the filter shifts by when sliding across the input. A stride of 1 means the filter moves by 1 pixel at a time, while a larger stride speeds up the convolution but reduces the output size. Small strides are used to detect fine-grained, precise features, while larger strides focus on high-level, coarse features.

Padding Pitfalls

One issue that arises with convolution is that the output size keeps shrinking with successive convolutions. For a 7×7 input convolved with a 3×3 filter, the output is 5×5. Convolve that again and you get a 3×3 output.

Not only does this limit the depth of CNNs you can build, but it also means that pixels around the edges are used much less in the output than pixels in the middle that have full neighborhood coverage by the filter.

The solution is to use zero padding, which means adding extra pixels around the borders of the input, filled with 0 values, before doing the convolution. This allows us to preserve the spatial dimensions and give edge pixels more influence. The most common types are:

• Valid convolution – no padding, output size is reduced
• Same convolution – add padding until output size equals input size

The amount of padding for "same" convolution that ensures the output size matches the input size is:

P = (H_f – 1) / 2

So for a 3×3 filter, we add 1 pixel of zero padding around the input. For a 5×5 filter, we add 2 pixels.

Going RGB

So far we‘ve looked at 2D convolution, but what about color images? An RGB image actually has three channels – red, green and blue – stacked on top of each other. So a 256×256 RGB image is really 256x256x3.

To convolve on RGB images, we use 3D filters of size H_fxW_fxC_f where the depth dimension C_f must equal the number of channels in the input (3 for RGB). 3D convolution works by doing 2D convolutions in parallel on each color channel and then summing the results together into a single 2D output.

Here‘s a visualization of 3D convolution:

Visualization of 3D convolution on an RGB image. Source: https://www.researchgate.net/figure/Convolution-layer-of-a-CNN-38_fig3_336805509

The ability to use 3D filters is what makes CNNs so powerful on color image data compared to prior approaches. The filters can learn color-specific features and how patterns combine across channels, like yellow circles or blue lines.

Multiple Filters = Multiple Features

In practice, instead of using a single filter, CNNs use multiple filters in parallel to detect many features simultaneously. Each filter has its own set of weights that are learned during training.

The number of filters is another hyperparameter that determines the depth of the output feature map. For example, an RGB input convolved with 64 filters of size 3x3x3 would produce a feature map of size H_oxW_ox64. You can think of each filter as a neuron, with the weights as the parameters of that neuron tuned to detect a specific feature.

Stacking multiple convolutions with many filters builds up hierarchical feature representations, from simple low-level features like edges to more complex high-level features like objects and scenes. Deep CNNs can have dozens or hundreds of convolutional layers, learning incredibly rich and sophisticated patterns.

Pool Party

In between convolutional layers, CNNs commonly include pooling layers. Pooling progressively reduces the spatial dimensions of the feature maps, lowering the amount of parameters and computation required. It also helps make feature representations more robust by providing some translation invariance.

The two main types of pooling are:

• Max Pooling – returns the maximum value from the portion of the feature map covered by the filter
• Average Pooling – returns the average of all values from the portion of the feature map covered by the filter

Here‘s an illustration of max pooling and average pooling:

Illustration of max pooling and average pooling. Source: https://computersciencewiki.org/index.php/Max-pooling_/_Pooling

Pooling acts as a kind of downsampling, showing if a feature is present in a region of the image, but not precisely where. It provides spatial invariance – even if the feature isn‘t perfectly aligned, the pooled result will be the same. This helps CNNs generalize better and be less sensitive to small shifts and distortions.

However, pooling does throw away a lot of information. Many state-of-the-art CNN architectures like ResNets and EfficientNets use very minimal or no pooling layers. Instead, they increase stride in the convolutions to reduce spatial dimensions gradually while learning. Pooling remains an important tool, but isn‘t always necessary.

Putting It All Together

A complete CNN architecture generally follows a common pattern of stacking multiple convolution + pooling layers to learn feature hierarchies, followed by a few fully-connected layers for classification or regression on the learned features.

Here‘s an example of the famous LeNet-5 CNN architecture:

The LeNet-5 CNN architecture. Source: https://engmrk.com/lenet-5-a-classic-cnn-architecture/

The specific number, size, stride, and connectivity of the layers are hyperparameters that are experimented with to find the optimal configuration for a given dataset and task. CNN architectures have evolved rapidly, from LeNet to AlexNet, VGGNet, GoogLeNet, ResNet, and beyond, each introducing new ideas and improving performance.

Summary

• CNNs are a powerful type of neural network designed for processing grid-like data such as images
• Convolution is the core operation, sliding filters across the input to detect features
• Filters are the building blocks of CNNs, learning to extract visual patterns
• Output size from convolution depends on input size, filter size, stride, and padding
• Zero padding helps control the spatial dimensions and give more influence to edge pixels
• 3D filters are used for RGB images, doing 2D convolutions on each channel
• Multiple filters are used to detect many features in parallel
• Pooling provides spatial invariance and downsampling, but isn‘t always necessary
• A typical CNN stacks convolution, pooling, and fully-connected layers to learn hierarchical features

CNNs have become the dominant approach for nearly all computer vision tasks. Understanding how convolution and filters work is key to designing and training effective CNNs. We‘ve only scratched the surface in this post – modern CNN architectures incorporate many other ideas like skip connections, depthwise separable convolutions, attention, and more. But convolution remains the core building block that enables the hierarchical feature learning that makes CNNs so powerful.