Must Know Questions for Deep Learning: An AI/ML Expert‘s Perspective

Deep learning has revolutionized the fields of computer vision, natural language processing, and many other domains in artificial intelligence in the last decade. As a data scientist or machine learning engineer, it is critical to have a solid understanding of the fundamental concepts behind deep neural networks. Interviews for deep learning positions at top tech companies often dive into the details of architectures, training algorithms, and practical considerations. In this post, we‘ll cover some of the most important deep learning questions you should be prepared to answer as an AI/ML expert, whether you‘re interviewing for a role or looking to apply deep learning to your own projects.

Neural Network Fundamentals

Let‘s start with the building block of all neural networks – the artificial neuron. Inspired by biological neurons in the brain, an artificial neuron takes a vector of inputs $\mathbf{x}$, multiplies each input by a weight $w_i$, sums them together with a bias term $b$, and then applies a non-linear activation function $f$ to produce an output $y$. Mathematically, this is represented as:

$$
y = f\left(\sum_{i} w_i x_i + b\right)
$$

Some common activation functions include:

• Sigmoid: $\sigma(x) = \frac{1}{1 + e^{-x}}$, squashes output to $(0, 1)$
• Hyperbolic tangent: $\tanh(x) = \frac{e^x – e^{-x}}{e^x + e^{-x}}$, squashes output to $(-1, 1)$
• Rectified Linear Unit (ReLU): $\text{ReLU}(x) = \max(0, x)$

Each has its own properties and benefits. Sigmoids are easy to interpret as probabilities but can saturate and kill gradients. Tanh functions are zero-centered which helps gradient flow. ReLUs are computationally efficient and help alleviate the vanishing gradient problem but can "die" if the learning rate is too high.

Stacking neurons together in multiple layers allows neural networks to learn complex non-linear functions. A single-layer perceptron has an input layer directly connected to an output layer and can only learn a linear decision boundary. A multi-layer perceptron (MLP) has one or more hidden layers in between, allowing it to model arbitrary functions given enough neurons.

A multi-layer perceptron with 2 hidden layers. Image credit: Wikipedia

Training Neural Networks

To actually learn the weights of a neural network, we typically use an optimization algorithm like mini-batch stochastic gradient descent (SGD). SGD works by repeatedly estimating the gradient $\nabla_\theta J(\theta)$ of the loss function $J$ on small batches of data, taking a step in the direction of the negative gradient to minimize the loss, and updating the weights $\theta$ using a learning rate hyperparameter $\alpha$:

$$
\theta_{t+1} = \thetat – \alpha \nabla\theta J(\theta_t)
$$

Backpropagation is the algorithm used to efficiently calculate the gradients by recursively applying the chain rule from the output layer back to the input layer. For a mini-batch of $m$ examples, the forward pass for an MLP with $L$ layers, weight matrices $W^{[l]}$, bias vectors $b^{[l]}$, and activation function $g$ is:

$$
\begin{aligned}
A^{[0]} &= X \
Z^{[l]} &= W^{[l]} A^{[l-1]} + b^{[l]} \
A^{[l]} &= g(Z^{[l]})
\end{aligned}
$$

The loss $J$ is computed between the predictions $\hat{y} = A^{[L]}$ and true labels $y$. The backpropagation pass then calculates the gradient of the loss w.r.t. each parameter:

$$
\begin{aligned}
dZ^{[L]} &= \frac{1}{m} (\hat{y} – y) \odot g‘(Z^{[L]}) \
dW^{[L]} &= dZ^{[L]} A^{[L-1]T} \
db^{[L]} &= \frac{1}{m} np.sum(dZ^{[L]}, \text{axis=1, keepdims=True}) \
dA^{[L-1]} &= W^{[L]T} dZ^{[L]}
\end{aligned}
$$

Here $\odot$ denotes element-wise multiplication and the gradients for the earlier layers are calculated recursively with $dZ^{[l]} = dA^{[l]} \odot g‘(Z^{[l]})$. Python pseudocode for training an MLP is:

# Forward pass to calculate predictions and loss
preds = model(X) 
loss = loss_fn(preds, y)

# Backward pass to calculate gradients
loss.backward()

# Update weights using SGD  
for param in model.parameters():
  param -= learning_rate * param.grad

While vanilla SGD takes a step in the direction of the negative gradient $-\nabla_\theta J(\theta_t)$ on each batch, other optimization algorithms adapt the learning rate for each parameter based on various heuristics to speed up convergence:

• Momentum: Moving average of gradients to dampen oscillations
• Adagrad: Larger updates for infrequent parameters, smaller updates for frequent parameters
• RMSProp: Adapts Adagrad to work well for non-convex loss surfaces by using an exponentially weighted moving average of squared gradients
• Adam: Combines momentum and RMSProp, generally a good default choice

"Adam is quite robust and well-suited to a wide range of non-convex optimization problems in the context of machine learning. Insofar, it is the best general-purpose optimizer." (Kingma & Ba, 2015)

There are several important considerations when training deep neural networks:

• Initialization: The initial values for the weights can have a big impact on convergence. Common heuristics are Xavier initialization (keeps variance of activations and backpropagated gradients roughly the same across layers) and He initialization (accounts for ReLU non-linearity).

• Learning rate schedules: Instead of using a fixed learning rate, gradually decaying the learning rate over time can improve convergence. Step decay drops the learning rate by a factor every few epochs. Cosine annealing smoothly decreases the learning rate following a cosine curve.

• Regularization: To combat overfitting, $L^1$ or $L^2$ regularization adds a penalty term to the loss function based on the magnitude of the weights. Dropout randomly drops a fraction (e.g. 50%) of neurons during training, forcing the network to learn redundant representations. Batch normalization re-centers and re-scales activations to stabilize training. Gradient clipping limits the size of gradients to avoid exploding gradients.

• Monitoring: It‘s important to track not just the training loss, but also metrics like validation accuracy to diagnose issues like overfitting and tune hyperparameters accordingly. Visualizing the loss landscape can provide insights into whether the model has reached a good optimum.

Example learning curves showing validation accuracy plateauing due to overfitting. Image credit: fast.ai

Convolutional Neural Networks

For structured data like images, convolutional neural networks (CNNs) have been the dominant approach since AlexNet‘s breakthrough performance on the ImageNet classification benchmark in 2012. The key building blocks of CNNs are convolutional layers and pooling layers:

• Convolutional layers: A set of learnable filters slide across the input, performing element-wise multiplications and summing the results to produce a feature map. This allows the network to learn spatial hierarchies of features, from simple edges to complex objects.

• Pooling layers: Downsample the feature maps by taking the max or average value in a sliding window, providing translation invariance and reducing the number of parameters.

Typical CNN architecture with alternating convolution and pooling layers followed by fully connected layers for classification. Image credit: Wikipedia

Through clever architecture designs like skip connections in ResNets and depthwise separable convolutions in MobileNets, CNNs have achieved impressive accuracy on large-scale image classification tasks while remaining computationally efficient.

Progress of ImageNet top-5 accuracy over time. Image credit: Papers With Code

Transfer learning with pre-trained CNNs has also become a powerful paradigm – by using a CNN like ResNet trained on a large dataset like ImageNet as a feature extractor and fine-tuning the last few layers on a target dataset, even small datasets can achieve strong performance. More recent work has demonstrated the potential of zero-shot transfer with architectures like CLIP and ALIGN that learn visual concepts from natural language supervision.

Recurrent Neural Networks

For sequence data like text or time series, recurrent neural networks (RNNs) are a natural fit as they can process variable-length sequences and maintain a "state" or memory over time. An RNN performs the same computation at each time step – updating its hidden state $h_t$ based on the current input $xt$ and previous hidden state $h{t-1}$:

$$
\begin{aligned}
ht &= \tanh(W{hh} h{t-1} + W{xh} x_t) \
yt &= W{hy} h_t
\end{aligned}
$$

However, standard RNNs struggle to learn long-term dependencies due to the vanishing/exploding gradient problem. Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) architectures address this by introducing learnable gates that control the flow of information into and out of the hidden state. An LSTM cell contains an input gate $i$, forget gate $f$, output gate $o$, candidate memory $\tilde{C}_t$ and memory cell $C_t$ and hidden state $h_t$ that are updated at each time step $t$ by:

$$
\begin{aligned}
i_t &= \sigma(xt W{xi} + h{t-1} W{hi} + b_i) \
f_t &= \sigma(xt W{xf} + h{t-1} W{hf} + b_f) \
o_t &= \sigma(xt W{xo} + h{t-1} W{ho} + b_o) \
\tilde{C}_t &= \tanh(xt W{xC} + h{t-1} W{hC} + b_C) \
C_t &= ft \odot C{t-1} + i_t \odot \tilde{C}_t \
h_t &= o_t \odot \tanh(C_t)
\end{aligned}
$$

Sequence-to-sequence RNNs with an encoder network that processes the input sequence and decoder network that generates the output sequence achieved breakthrough results on tasks like machine translation, aided by the attention mechanism which learns to align the relevant encoder hidden states to the decoder hidden state at each time step.

In recent years though, Transformer architectures based solely on attention mechanisms have largely replaced RNNs as the state-of-the-art for sequence modeling tasks. Transformers are more efficient to train on modern hardware and have enabled training on web-scale corpora, unlocking powerful language models like GPT-3, PaLM, and Chinchilla that can perform few-shot learning on a wide range of tasks through prompting.

Practical Deployment Considerations

Deploying deep learning models in production environments presents a number of engineering challenges beyond training accurate models:

• Model efficiency: Since GPUs/TPUs can be expensive, techniques like quantization (storing weights with fewer bits), pruning (removing redundant weights), and knowledge distillation (training a smaller student model to mimic a larger teacher) are used heavily in practice to reduce model size and latency.

• Serving infrastructure: Frameworks like TensorFlow Serving, NVIDIA Triton, Seldon, and cloud platforms like AWS SageMaker, Google Vertex AI, etc provide scalable solutions for hosting deep learning models behind an API endpoint. Considerations include how to do efficient batching, autoscaling, load balancing, and monitoring.

• Monitoring: In addition to standard software metrics like latency, throughput, and error rates, it‘s critical to track the live performance of deep learning models, as data drift and concept drift can cause silent failures. Explainability techniques like saliency maps and neuron activation analysis can help debug issues.

Conclusions

Deep learning remains a rapidly evolving field, with state-of-the-art architectures like transformer language models achieving remarkable results on benchmarks like SuperGLUE and even generating code from natural language. As an AI/ML practitioner, staying up to date with the latest techniques is both exciting and challenging.

While this post covered many of the core concepts, there is still much more to learn – from the mathematical foundations of representation learning to the systems design of deep learning infrastructure. Let me know in the comments what other topics you‘d like to see covered and what you found most useful here!