Why GPUs are Powering the Deep Learning Revolution

Deep learning has transformed the field of artificial intelligence in recent years, enabling machines to achieve superhuman performance on tasks ranging from image classification and speech recognition to playing complex games like Go. However, training the large and sophisticated neural networks that power these breakthroughs requires an enormous amount of computational power.

While deep neural networks can be run on standard CPUs, the game-changing results of the last decade have been made possible by a different class of hardware—graphics processing units, or GPUs. Originally developed to accelerate 3D graphics rendering, GPUs have proven uniquely well-adapted to the demands of deep learning workloads.

The Performance Advantage of GPUs

To understand why GPUs have become so crucial for deep learning, it‘s instructive to compare their performance characteristics to traditional CPUs. The following table shows the key specs of NVIDIA‘s V100 GPU compared to two Intel Xeon CPUs [1]:

The performance advantage of the V100 is staggering—in FP16 precision, it provides nearly 40X the throughput of the best CPU. This is enabled by the massive parallelism of the GPU‘s architecture. With 5120 CUDA cores, the V100 can execute tens of thousands of threads simultaneously.

The V100 also holds a significant memory bandwidth advantage. Maximizing the rate of data transfer between memory and compute cores is crucial for keeping the GPU‘s arithmetic units fully utilized. With 900 GB/s of bandwidth enabled by closely integrating high-bandwidth memory (HBM) on the same package as the GPU, the V100 can feed its cores much faster than a CPU limited by off-chip memory.

GPUs in Action: Landmark Results

This raw performance translates into dramatically accelerated training times and the ability to develop significantly larger and more sophisticated models. Some landmark examples of GPU-powered breakthroughs include:

• In 2020, OpenAI trained GPT-3, a language model with 175 billion parameters, on 1024 NVIDIA V100 GPUs [2] for several months. This produced an unprecedented leap in natural language processing capability.

• AlphaFold 2, DeepMind‘s AI system for protein folding, was trained on 128 TPU v3 cores (which provide similar matrix multiplication performance to V100 GPUs) and took a few weeks [3]. It achieved a stunning breakthrough with potential to revolutionize biology and drug discovery.

• MuZero, a reinforcement learning model also developed by DeepMind, achieved superhuman performance across Go, chess, and shogi [4]. It was trained on 1000 TPUs for several weeks.

The sheer scale of computation enabled by GPU acceleration has expanded the frontier of what‘s possible with deep learning. Models with billions or even trillions of parameters can now be contemplated.

Key Enablers of GPU Deep Learning

Several key technologies have enabled GPUs to be effectively applied to deep learning workloads:

• CUDA: NVIDIA‘s parallel computing platform allows developers to harness the power of GPUs using familiar programming languages. Higher level libraries built on CUDA abstract away many complexities.

• cuDNN: NVIDIA‘s deep learning primitives library provides highly optimized building blocks for neural networks, including convolution, pooling, normalization, and activation functions. Frameworks like TensorFlow and PyTorch achieve much of their performance by invoking cuDNN under the hood.

• Tensor Cores: Introduced in the Volta GPU architecture, Tensor Cores are specialized units that dramatically accelerate the mixed-precision matrix math at the heart of deep learning. Tensor Cores exploit the insight that deep neural networks can be trained with lower precision (e.g. FP16) without sacrificing accuracy. Tensor Cores also support new numeric formats tailored for deep learning like bfloat16 and TF32.

Model Parallelism and Data Parallelism

Modern neural networks often exceed the memory capacity of a single GPU. Splitting models across multiple GPUs has therefore become a necessity. There are two primary ways this is done:

1. Data parallelism: Multiple copies of the model are trained on different subsets of the data in parallel. Gradients are averaged between models periodically to keep them in sync.

2. Model parallelism: A single model is split across multiple GPUs, with each GPU responsible for storing and computing a subset of the model‘s layers.

In practice, a combination of data and model parallelism is often used to scale training to many GPUs. This allows researchers to consider massive models with compute and memory requirements far beyond what is feasible on a single device.

The GPU Competitive Landscape

NVIDIA has long held a dominant position in the market for GPUs in deep learning. This is thanks to their significant investment in deep learning-specific hardware (e.g. Tensor Cores) and software (CUDA, cuDNN, etc).

In recent years, NVIDIA released the A100 GPU based on the Ampere architecture. The A100 doubled peak throughput compared to the V100, reaching 312 TFLOPS for FP16 [5]. It also introduced new Tensor Core formats like TF32 which provides FP32-equivalent precision at dramatically higher throughput.

NVIDIA‘s most recent H100 GPU based on the Hopper architecture further pushes the envelope, offering up to 2000 TFLOPS of FP8 precision [6]. This focus on low-precision, high-throughput formats tailored for deep learning reflects the evolving nature of the field.

While NVIDIA remains the leader, other players are actively vying for a piece of the rapidly growing AI accelerator market. AMD‘s MI200 GPU offers competitive FP16 and FP32 performance and introduces support for new formats like FP24 [7]. Intel‘s newly launched Data Center GPU Max Series will also bring serious competition with over 100 TFLOPS of FP16 [8].

Meanwhile, several startups are developing AI-specific processors that depart from the traditional GPU architecture. These include Cerebras‘ Wafer Scale Engine [9] and SambaNova‘s Reconfigurable Dataflow Unit [10]. Cloud hyperscalers like Google and Amazon are also fielding custom AI chips (TPUs and Inferentia, respectively).

The Future of AI Computing

The next decade will see AI continue to advance at a blistering pace, transforming industries and unlocking new capabilities. Sustaining this progress will require ongoing innovations in computing hardware and software.

For the foreseeable future, GPUs are likely to remain the workhorses for training state-of-the-art deep learning models in the cloud and data center. Their unique combination of high throughput, parallelism, and programmability makes them ideally suited for this role. Ongoing architectural evolutions (e.g. continuing to scale Tensor Cores) will further cement this position.

At the same time, deploying trained models efficiently on power and thermally constrained edge devices will require more specialized inference accelerators. Here, we can expect a more diverse range of processors optimized for specific use cases – for example, Apple‘s Neural Engine for mobile devices [11] or Tesla‘s FSD chip for autonomous vehicles [12].

More broadly, the future of AI computing will be heterogeneous – with GPUs working in concert with CPUs, AI accelerators, and other domain-specific hardware to provide balanced and efficient systems. Unified programming models and software stacks that can abstract away these complexities and allow developers to seamlessly leverage diverse compute assets will be key.

As AI pushes further into every facet of the economy, organizations‘ ability to harness these technologies will become a key competitive differentiator. Strategic investment in hardware and software capabilities – with GPU expertise as a critical pillar – will be essential. We have only begun to scratch the surface of what deep learning can achieve, and GPUs will continue to light the way forward.