A Deep Dive into Separable Convolutions: Foundations, Architectures, and Efficient Implementation

Introduction

Convolutional neural networks (CNNs) have become the dominant approach for visual perception tasks, achieving state-of-the-art results on challenges like image classification, object detection, segmentation, and more. The power of CNNs stems from their ability to learn rich, hierarchical representations of visual data through the use of convolutional layers that scan learned filters across the input.

However, the representational capacity of CNNs comes with a significant computational cost, which can limit their practical applicability in resource-constrained environments like mobile devices or real-time systems. Much research has focused on improving the efficiency of CNNs while maintaining high accuracy.

Separable convolutions have emerged as a key technique for building lightweight, efficient CNN architectures. By factoring the standard convolution operation into separate spatial and depth-wise components, separable convolutions can drastically reduce the computational burden while retaining strong performance.

In this post, we‘ll dive deep into the details of separable convolutions, exploring their mathematical formulation, practical implementation, and use in modern CNN architectures. We‘ll examine the tradeoffs involved and provide guidance on when and how to leverage separable convolutions effectively. While separable convolutions are not a silver bullet, understanding them is valuable for anyone working to develop efficient models for deployment at the edge.

The Computational Burden of Regular Convolutions

To appreciate the value of separable convolutions, it‘s important to understand the computational costs associated with regular convolutions and how these scale with the size of the input data and model.

Consider a convolutional layer with the following properties:

• Input tensor size: $D_i \times H_i \times W_i \times C_i$
• Output tensor size: $D_o \times H_o \times W_o \times C_o$
• Kernel size: $D_k \times H_k \times W_k$

where $D$ is the depth (number of channels), $H$ is the height, $W$ is the width, and the input and output tensors have $C_i$ and $C_o$ channels respectively.

The number of parameters in this layer is:

$D_k \times H_k \times W_k \times C_i \times C_o$

And the number of floating-point operations (FLOPs) required is:

$D_k \times H_k \times W_k \times C_i \times H_o \times W_o \times C_o$

We can see that the computational cost scales linearly with the spatial dimensions of the kernel and quadratically with the number of input and output channels. For concreteness, let‘s consider some realistic values:

• Input tensor: 32 x 56 x 56 x 128 (batch size 32, 56×56 feature maps, 128 channels)
• Output tensor: 32 x 28 x 28 x 256 (28×28 feature maps, 256 channels)
• Regular 3D convolution with 3 x 3 x 3 kernel, stride 2

This convolution has:

• Parameters: 3 3 3 128 256 = 884,736
• FLOPs: 3 3 3 128 28 28 256 = 924,844,032

That‘s nearly 1 million parameters and 1 billion FLOPs for a single layer! It‘s clear that such high costs can rapidly become burdensome, especially for deep networks. This motivates the development of techniques to improve the efficiency of convolutions.

Depthwise Separable Convolutions

The key insight behind separable convolutions is that we can factor the standard 3D convolution into two separate operations:

1. Depthwise convolution: Perform lightweight spatial convolutions independently for each input channel
2. Pointwise convolution: Use $1 \times 1$ convolutions to merge the information across channels

Mathematically, if the input tensor is $X$ and the output tensor is $Y$, a depthwise separable convolution with kernel $K$ can be written as:

$Y = K{1 \times 1} * (K{D_k \times H_k \times W_k} \otimes X)$

where $\otimes$ denotes the depthwise convolution operation and $*$ denotes the pointwise convolution. $K_{D_k \times H_k \times Wk}$ is the depthwise kernel (one per input channel) and $K{1 \times 1}$ is the pointwise kernel.

Figure 1 illustrates the steps involved:

Figure 1: Illustration of depthwise separable convolution. (Image credit: Bendersky, 2018)

Let‘s walk through a concrete example to see how this works and analyze the computational savings. Consider the same example convolution from the previous section, but implemented as a depthwise separable convolution with:

• Depthwise kernel: 3 x 3 x 3 (applied independently to each input channel)
• Pointwise kernel: 1 x 1 x 128 x 256

The depthwise step requires:

• Parameters: 3 3 3 * 128 = 3,456
• FLOPs: 3 3 3 128 28 * 28 = 3,612,672

And the pointwise step requires:

• Parameters: 1 1 128 * 256 = 32,768
• FLOPs: 1 1 128 28 28 * 256 = 25,690,112

In total, the depthwise separable version has:

• Parameters: 3,456 + 32,768 = 36,224
• FLOPs: 3,612,672 + 25,690,112 = 29,302,784

Compared to the regular convolution, this is a 24x reduction in parameters and a 31x reduction in FLOPs! Table 1 summarizes the comparison:

Convolution Type	Parameters	FLOPs
Regular Convolution	884,736	924,844,032
Depthwise Separable	36,224	29,302,784

Table 1: Comparison of computational cost for regular vs depthwise separable convolutions.

The separable version achieves these savings by decomposing the convolution operation and applying the spatial kernels separately for each channel before merging the results. This is much more efficient than learning a full 3D kernel operating jointly over all channels.

Of course, the reduction in parameters and FLOPs comes with a potential tradeoff in representational capacity, since the separable convolution has fewer degrees of freedom. However, in practice, many models can achieve comparable accuracy with separable convolutions while being much more efficient. Striking the right balance between accuracy and efficiency is key.

Efficient CNN Architectures with Depthwise Separable Convolutions

Depthwise separable convolutions have been used as a building block in many modern efficient CNN architectures. Some prominent examples include:

• Xception (Chollet, 2017): This architecture is based entirely on depthwise separable convolutions, which replace the regular convolutions in an Inception-style network. Xception outperforms Inception V3 on ImageNet classification while being more efficient.

• MobileNet (Howard et al., 2017): Designed for mobile and embedded vision applications, MobileNet uses 3×3 depthwise separable convolutions followed by pointwise convolutions as its main building block. MobileNet achieves comparable accuracy to VGG-16 on ImageNet with only 1/30 the computational cost and model size.

• MobileNetV2 (Sandler et al., 2018): An enhancement of MobileNet that introduces inverted residual blocks and linear bottlenecks. These improve the expressiveness of the depthwise separable convolutions and make the model more accurate and efficient.

• MnasNet (Tan et al., 2019): A model architecture discovered via neural architecture search techniques, optimized for both accuracy and latency. MnasNet uses depthwise separable convolutions in a novel "inverted bottleneck" module and outperforms MobileNetV2 and NASNet on ImageNet while being 1.6-3.5x faster.

To see the practical impact, let‘s compare a ResNet-50 model (which uses regular convolutions) to an Xception model on ImageNet classification:

Model	Top-1 Acc.	Params	FLOPs
ResNet-50	76.0%	25.6M	3.86G
Xception	79.0%	22.9M	4.23G

Table 2: ImageNet classification performance of ResNet-50 vs Xception. Data from Chollet, 2017.

Despite having fewer parameters, Xception outperforms ResNet-50 while using a similar number of FLOPs. This demonstrates that depthwise separable convolutions can be used to build models that are both highly accurate and efficient.

Similar comparisons with other architectures like MobileNet and MnasNet further validate the effectiveness of depthwise separable convolutions as a tool for efficient CNN design. However, it‘s important to note that while separable convolutions are very useful, they are typically used in conjunction with other techniques like inverted residuals, squeeze-and-excitation modules, neural architecture search, and others to achieve state-of-the-art results.

Variants and Enhancements

Since their introduction, several variants and enhancements to depthwise separable convolutions have been proposed to further improve their expressiveness and efficiency. Some examples include:

• Inverted residuals (Sandler et al., 2018): A modification where the bottleneck layer uses a higher-dimensional representation than the inputs, improving accuracy in MobileNetV2.

• Grouped pointwise convolutions (Zhang et al., 2018): Splitting the pointwise convolution into groups and shuffling the channels, enabling even more efficient models like ShuffleNet.

• Dilated separable convolutions (Wang et al., 2018): Incorporating dilation to increase the receptive field without increasing parameters.

• Learned group convolutions (Huang et al., 2018): Learning the optimal grouping strategy for the pointwise convolution to better capture channel-wise dependencies.

Many state-of-the-art efficient CNN architectures employ these and other enhancements together with the core depthwise separable convolution building block. Active research continues in this area to further optimize the tradeoffs between accuracy, latency, power usage, and model size.

Practical Guidance and Conclusion

We‘ve seen that depthwise separable convolutions are a powerful tool for building efficient CNN architectures that can run effectively on resource-constrained devices. However, it‘s important to apply them judiciously and understand the tradeoffs involved.

Some key considerations when using separable convolutions:

• Separable convolutions tend to work better in deeper architectures where the savings in parameters and FLOPs can compensate for the reduction in representational capacity of any single layer. Very shallow separable models may underfit.

• The efficiency gains of separable convolutions are most significant for layers with large kernel sizes (e.g. 5×5 or 7×7) and high-dimensional inputs/outputs. The overhead may not be worthwhile for small 1×1 layers.

• Separable convolutions often require more regularization (like batch normalization and dropout) and may take longer to converge due to the limited capacity of each layer.

• It‘s common to employ a mix of separable and regular convolutions (e.g. using regular convs in the early layers and separable convs in the later layers).

• Separable convolutions can be combined with techniques like inverted residuals, squeeze-and-excitation modules, and neural architecture search to further boost accuracy and efficiency.

Despite these considerations, depthwise separable convolutions are an essential component of the modern efficient CNN toolbox. Understanding how they work and when to use them is valuable for anyone working on deploying high-performance computer vision models on the edge.

As the field continues to advance, we can expect to see further innovations in efficient CNN design that build upon the key insights of separable convolutions. It‘s an exciting area of research with significant real-world impact.

Hopefully this deep dive has given you a comprehensive understanding of separable convolutions and their role in efficient CNN architectures. For further reading, check out some of the papers referenced throughout the post. Happy efficient modeling!