CPUs vs GPUs for Machine Learning: A Comprehensive Guide

Introduction

Machine learning (ML) and artificial intelligence (AI) workloads have fundamentally transformed computing – demanding unprecedented levels of parallelism and compute density. Central processing units (CPUs) and graphics processing units (GPUs) are the two most prevalent classes of processors used for ML today.

Understanding the key properties and differences of these computational engines is critical for data scientists and ML engineers to design efficient software and hardware stacks. In this article, we‘ll take a deep dive into the architectures and performance characteristics of CPUs and GPUs. We‘ll examine the suitability of each for different ML workloads, from model training to inference. Finally, we‘ll look at future trends and emerging technologies for AI acceleration.

CPU Architecture and Characteristics

CPUs are the heart of every computer system, executing the widest diversity of workloads from operating systems to databases. Modern CPUs are designed for general purpose computation with complex control logic and large low-latency caches to efficiently switch between tasks.

A CPU consists of several key components:

• Arithmetic logic unit (ALU): executes arithmetic and logic operations
• Control unit: directs instructions to be executed
• Registers: provide fast storage for a small amount of data
• Cache: larger low-latency memory to store frequently accessed data
• DRAM controller: interfaces with main memory

Simplified architecture of a modern CPU. Source: ExtremeTech

CPUs typically have a modest number of cores (2-24) that operate at high clock frequencies. Each core has a large amount of cache and complex branch prediction logic to minimize latency. To improve efficiency, modern CPUs also implement extensions like Intel AVX-512 and ARM SVE that perform single-instruction-multiple-data (SIMD) operations.

While CPUs excel at quickly executing diverse tasks, they are not optimized for the massively parallel matrix computations that form the backbone of modern ML.

GPU Architecture and Characteristics

In contrast to CPUs, graphics processing units (GPUs) specialize in throughput-oriented workloads with thousands of cores optimized for parallel computation. Originally designed for rendering computer graphics, GPUs have found new life as accelerators for scientific computing and machine learning.

The key architectural difference with GPUs is how transistors are allocated. Instead of complex control logic and caches, GPU transistors are primarily devoted to arithmetic logic units (ALUs). This allows GPUs to execute thousands of threads in parallel.

GPUs are organized into several streaming multiprocessors (SMs), each with hundreds of cores that share control logic and cache. Each SM has a large register file, shared memory, and L1/texture cache that enable low latency access to data.

Simplified architecture of an NVIDIA GPU streaming multiprocessor (SM). Source: NVIDIA

To maximize utilization, GPUs implement fine-grained multithreading. Threads are organized into warps of 32 that execute in lockstep (SIMT). Warps are further grouped into thread blocks that can communicate via shared memory and synchronize execution. This hierarchy maps well to machine learning primitives like matrix multiplication.

When comparing raw theoretical performance, high-end GPUs can achieve over 312 teraFLOPS of mixed-precision compute and over 2 TB/s of memory bandwidth – more than an order of magnitude higher than leading CPUs.

Machine Learning Training Workloads

Training a deep neural network involves iteratively updating the weights of the model to minimize a loss function over millions of training examples. This process is computationally intensive, requiring up to 10^17 floating point operations to train modern language models like GPT-3.

GPUs have several properties that make them ideal for accelerating ML training:

• Massive parallelism: Weight updates can be parallelized across neurons and training examples. GPUs excel at large matrix multiplications that form the core of backpropagation.
• Large memory bandwidth: Training large models requires storing millions of parameters and activations. GPUs have an order of magnitude more bandwidth to shuttle data.
• Mixed precision: Reduced precision math like FP16 or BF16 is often sufficient for training. NVIDIA tensor cores provide massive speedups for mixed-precision matrix math.

Convolutional neural networks (CNNs) used in computer vision and transformer language models have seen the largest gains from GPU acceleration. The regular structure of these networks maps well to the SIMT architecture of GPUs.

For very large models, training is distributed across many GPUs. Data parallelism techniques like HorovodRunner allow efficient synchronization of gradients across GPUs. Model parallelism splits layers of the network across different GPUs or nodes.

Machine Learning Inference Workloads

While training a model is done once, inference (or prediction) is typically done repeatedly in production. Inference has much lower computational demands than training, often in the order of a few GFLOPS per example. However, inference in production requires strict latency targets, often in the millisecond range.

CPUs were traditionally used for inference due to their low latency. However, GPUs have made significant strides in inference performance:

• Concurrent streams: GPUs can run multiple inference streams in parallel, maximizing utilization.
• Reduced precision: INT8 or FP16 precision is usually sufficient for inference, allowing tensor cores to further accelerate performance.
• Frameworks: TensorRT and ONNX Runtime allow optimized inference pipelines on GPUs.
• Memory: On-chip RAM and high-bandwidth caches reduce memory transfer overheads.

NVIDIA T4 GPUs are optimized for inference with tensor cores and up to 130 TOPS of INT8 performance. For very latency sensitive applications, NVIDIA TensorRT can compile models into CUDA kernels that run in a few milliseconds.

GPU Software Ecosystem for ML

The availability of mature software frameworks and libraries has accelerated the adoption of GPUs for ML. Key frameworks include:

• CUDA: NVIDIA‘s parallel computing platform and API that allows fine-grained control of GPU execution
• cuDNN: Optimized primitives for deep neural networks
• TensorFlow: Google‘s end-to-end open source platform for ML
• PyTorch: Facebook‘s Python-first ML framework
• MXNet: Apache ML framework emphasizing linear scalability

Popularity of deep learning frameworks over time. Source: Medium

Higher level ML platforms like Keras, Hugging Face, and fast.ai abstract many details of working with GPUs. Distributed learning frameworks like Horovod and PyTorch DDP simplify multi-GPU training.

Cloud providers like AWS, GCP, and Azure also offer managed services with GPUs for ML workflows. This can be attractive for bursty workloads or to avoid large capital expenditures.

Industry Adoption and Benchmarks

GPUs have seen rapid adoption in hyperscale data centers and public clouds. NVIDIA estimates that over 70% of the top 500 supercomputers in the world are now accelerated by GPUs. In 2019, OpenAI specifically designed a huge cluster of over 1,500 V100 GPUs to train GPT-2.

In the public cloud, Amazon AWS, Microsoft Azure, Google GCP and Oracle OCI all offer NVIDIA GPU instances. As of 2020, AWS offers instances with up to 16 V100 GPUs (p3dn.24xlarge) and 8 A100 GPUs (p4d.24xlarge), with over 600 GB/s of bandwidth.

To quantify the performance benefits of GPUs for ML, let‘s look at some benchmarks from MLPerf, an industry-standard suite for testing ML performance across a variety of hardware:

MLPerf training benchmarks comparing NVIDIA V100 GPU to Intel Xeon Platinum 8168 CPU.

On all three models, the V100 GPU shows dramatic speedups over CPUs, ranging from 25x-59x. These speedups come from the specialized tensor cores and large memory bandwidth of the V100. Distributed training across many GPUs provides near linear scaling.

Future Trends in AI Accelerators

With the insatiable computational demands of modern ML, there is an arms race to design the next generation of AI accelerators. Some key trends:

• Larger models: GPT-3 and other giant language models are trained on clusters of thousands of GPUs. Future GPU architectures will continue to focus on multi-GPU scalability.
• Heterogeneous computing: Many workloads require a mix of scalar, matrix, and spatial computations. Future accelerators may have specialized cores for each (e.g. NVIDIA Grace).
• Sparsity: ML models are often overparameterized with many weights close to zero. Exploiting weight and activation sparsity will be key to continued performance gains.
• Near-memory computing: Moving computation closer to memory can alleviate the memory bandwidth bottleneck. Techniques like HBM2 and processing-in-memory (PIM) will be critical.
• Open software stacks: RISC-V and the rise of open-source hardware may drive innovation in AI accelerators beyond a few incumbents. Projects like TVM and Apache TVM aim to provide open compiler stacks.
• Neuromorphic computing: For extremely low power inference, specialized neuromorphic architectures that emulate the brain may become viable. Examples include IBM TrueNorth and Intel Loihi.

Tech giants are investing heavily to develop custom silicon for AI. Google has deployed tensor processing units (TPUs) in its data centers. Startups like Cerebras, Graphcore, and SambaNova have raised hundreds of millions to challenge NVIDIA‘s dominance.

Conclusion

The rapid progress in ML models over the past decade is directly tied to the rise of GPU computing. GPUs have enabled training models with billions of parameters and lightning fast inference in production. For data scientists and ML engineers, understanding the performance characteristics of CPUs and GPUs is key to designing efficient ML pipelines.

At the same time, AI-specific accelerators continue to grow in sophistication. It‘s an exciting time as startups and tech giants alike race to design the next generation of AI chips.

With ML poised to transform industries from healthcare to robotics to scientific research, selecting the right mix of computational engines will only grow in importance. Here‘s to further innovations at the intersection of ML algorithms and computing substrates!