Feed Forward Neural Networks: Intuition on Forward Propagation

Feed forward neural networks are a foundational type of artificial neural network that have wide-ranging applications in machine learning. They are used for tasks like classification, regression, and function approximation. While the inner workings of neural networks can seem complex and mysterious at first, developing an intuition for the core processes can make them much more approachable. In this post, we‘ll focus on understanding the forward propagation process that lies at the heart of how feed forward networks operate.

Neural Network Basics

At a high level, a feed forward neural network can be thought of as a function that maps input data to outputs. They are called "feed forward" because information flows through the network in one direction, from the input layer through the hidden layers to the output layer. The goal is to learn the optimal mapping function from inputs to outputs based on training data.

Neural networks are composed of interconnected nodes called neurons, arranged in layers. Each neuron receives weighted inputs, computes an output value, and passes it to neurons in the next layer. This process of calculating neuron outputs and passing them forward is known as forward propagation.

The Forward Propagation Process

Let‘s walk through the forward propagation process step-by-step to build intuition for how it works. Consider a simple feed forward network with an input layer, one hidden layer, and an output layer.

Inputs and Weights

The input layer receives the input data, with each input neuron corresponding to an individual feature. Each connection between neurons has an associated weight that determines the strength and sign of the connection.

Weights are typically initialized randomly and then learned during training to minimize the network‘s loss on the training data. Intuitively, the weights determine how much influence each input has on a neuron‘s output.

Some common weight initialization strategies include:

• Xavier initialization: Weights are drawn from a distribution with variance based on the number of input and output neurons, which helps prevent vanishing or exploding gradients.
• He initialization: Similar to Xavier but uses a different variance calculation more suitable for ReLU activation functions.
• Random normal/uniform: Weights are drawn from a normal or uniform distribution with specified mean and variance.

The choice of initialization can impact the speed and stability of training, and is an active area of research.

Weighted Sum

Inside each neuron, the first step is to calculate the weighted sum of its inputs. For a neuron $j$ in layer $l$, the weighted sum $z$ is calculated as:

$z_j^l = \sumk w{jk}^l \cdot a_k^{l-1} + b_j^l$

where $w_{jk}$ is the weight from neuron $k$ in the previous layer $l-1$ to neuron $j$ in layer $l$, $a_k$ is the activation output from neuron $k$ in the previous layer, and $b_j$ is the bias term for neuron $j$. Essentially, each input is multiplied by its weight, these values are summed, and the bias is added.

The weighted sum represents a linear combination of the inputs. Intuitively, the weights are knobs that control the influence of each input on the neuron‘s output, while the bias allows shifting the output.

For example, consider a neuron with two inputs $x_1=2$ and $x_2=3$, with corresponding weights $w_1=0.5$ and $w_2=-1.0$, and a bias $b=1$. The weighted sum would be:

$z = 0.5 \cdot 2 + (-1.0) \cdot 3 + 1 = -1$

Activation Function

The weighted sum $z$ is then passed through a non-linear activation function $\sigma$ to get the final output $a$ of the neuron:

$a_j^l = \sigma(z_j^l)$

The activation function introduces non-linearity into the network, which is critical for learning complex functions. Without non-linearity, the network would just be a linear transformation of the inputs, no matter how many layers it had.

Some common activation functions include:

• Sigmoid: $\sigma(x) = \frac{1}{1+e^{-x}}$, squashes values to the range (0,1)
• Tanh: $\tanh(x) = \frac{e^x – e^{-x}}{e^x + e^{-x}}$, squashes values to the range (-1,1)
• ReLU: $\text{ReLU}(x) = \max(0,x)$, clips negative values to 0

Intuitively, the activation function allows the neuron to make a decision based on its inputs. It‘s analogous to a neuron "firing" or not based on whether its input crosses some threshold.

The choice of activation function can have a significant impact on the network‘s performance. ReLUs have become popular in recent years due to their simplicity and ability to alleviate the vanishing gradient problem in deep networks. However, there are many other options like Leaky ReLU, ELU, and Swish that can work better in certain situations.

In our example, if we apply the sigmoid activation to the weighted sum $z=-1$, we get:

$a = \sigma(-1) = \frac{1}{1+e^1} \approx 0.27$

So this neuron would output 0.27 given the inputs and weights.

Propagating through Layers

This weighted sum and activation function application process is repeated for each neuron in the hidden layer, generating a set of outputs. These then become the inputs for the next layer, and so on until we reach the output layer.

The output layer produces the network‘s predictions. For classification with $K$ classes, the output layer typically has $K$ neurons, each representing the model‘s predicted probability for that class. The softmax activation function is commonly used to ensure these probabilities sum to 1:

$\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_k e^{z_k}}$

For regression, the output layer may have just one neuron with a linear activation (i.e., no transformation of the weighted sum).

Representational Power and Depth

Through this forward propagation process, feed forward neural networks can learn to approximate very complex non-linear functions. The key is that each layer is performing a non-linear transformation of the inputs, and by composition of these non-linear transformations, a wide variety of functions can be represented.

The universal approximation theorem states that a feed forward network with just one hidden layer containing a finite number of neurons can approximate any continuous function to arbitrary precision, under mild assumptions on the activation function. However, the width of this hidden layer may need to be very large.

In practice, deep networks with many hidden layers tend to work better than shallow ones. The intuition is that the earlier layers can learn lower-level features, while later layers can combine these into higher-level representations. For example, in image recognition, early layers may detect edges and textures, middle layers may detect parts and objects, and later layers may detect full scenes.

Research has shown that deeper networks can be exponentially more efficient at representing certain classes of functions than shallow ones. The Inception and ResNet architectures, which achieved state-of-the-art performance on ImageNet, used dozens or even hundreds of layers.

However, very deep networks can be difficult to train due to vanishing/exploding gradients and the degradation problem. Techniques like residual connections, batch normalization, and careful initialization have helped alleviate these issues.

What Do Hidden Layers Learn?

A natural question is what the hidden layers of a trained network are actually learning. One way to visualize this is to look at the activation patterns of the neurons in response to different inputs.

For example, in a network trained on images, we can find the input images that maximally activate each neuron. If we do this for neurons in the first hidden layer, we often see that they respond to simple features like edges or colors. Neurons in higher layers tend to respond to more complex and abstract features.

Another approach is feature visualization, where we start with random noise and optimize the image to maximally activate a chosen neuron or set of neurons, using backpropagation. This can give insight into the preferred stimulus of a neuron.

Latest Developments and Applications

Feed forward neural networks continue to be a workhorse of deep learning, with active research yielding steady improvements. Some notable recent developments as of 2023:

• ViT (Vision Transformer) and MLP-Mixer architectures have shown that pure attention-based models using multi-layer perceptrons can match or exceed the performance of convolutional networks on image classification tasks.
• Techniques like self-supervised pre-training, transfer learning, and fine-tuning have made it possible to train very large models on huge datasets and then adapt them to specific tasks with less data.
• Neural architecture search has led to highly optimized, task-specific network designs.
• Efficient implementation frameworks and hardware have dramatically increased the speed and scale of training.

Some exciting recent applications include:

• AlphaFold 2, a deep network for protein structure prediction, has solved a 50-year grand challenge in biology. It uses a novel attention-based architecture and was trained on a large database of protein sequences and structures.
• Language models like GPT-3 and PaLM, with hundreds of billions of parameters trained on vast text corpora, have shown remarkable language understanding and generation capabilities. They use transformer architectures which are based on attention mechanisms.
• Image generation models like DALL-E 2 and Stable Diffusion can create highly realistic and creative images from textual descriptions. They typically combine convolutional networks, transformers, and variational autoencoders.
• Deep reinforcement learning has achieved superhuman performance in games like Go, Poker, and Dota 2, and is being applied to robotics, autonomous driving, and scientific discovery.

Conclusion

Feed forward neural networks are a powerful and widely-used tool in machine learning. Understanding the intuition behind the forward propagation process is key to demystifying how these networks can learn to solve complex problems.

By composing many non-linear transformations of the input data using learned weights and activation functions, feed forward networks can approximate a very wide range of functions. While there are more advanced architectures for certain types of data and tasks, feed forward networks remain an essential part of the deep learning toolkit.

As research progresses, we can expect to see continued improvements in the scale, efficiency, and performance of feed forward networks, as well as novel applications across diverse domains. Grasping the core principles behind them is a crucial step in the journey of becoming a proficient AI and machine learning practitioner.