A Deep Dive Into Neural Network Activation Functions

Activation functions are the secret sauce that enables neural networks to learn complex non-linear patterns in data. As a crucial component of nearly every neural network architecture, the choice of activation function has a major impact on model training dynamics and performance. In this in-depth guide, we‘ll review the full landscape of neural network activation functions, from classics like sigmoids and ReLUs to cutting-edge options like SELUs.

We‘ll provide expert guidance grounded in the latest AI research to help you choose the right activation function for your model and application. Along the way, we‘ll dive deep into the theory of activation functions, visualize their key properties, explore their biological basis, and highlight promising future research directions. Whether you‘re a deep learning researcher or practitioner, by the end of this guide you‘ll gain a solid understanding of these powerful and fascinating functions that are fueling advances in AI. Let‘s jump in!

The Role of Activation Functions

At its core, a neural network is a complex function composed of many simple building blocks called neurons. Each neuron computes a weighted sum of its inputs and then applies a non-linear activation function to generate its output. Mathematically, for a single neuron:

$$
y = f(w_1x_1 + w_2x_2 + … + w_nx_n + b)
$$

where $x_1, x_2, …, x_n$ are the inputs, $w_1, w_2, …, w_n$ are the weights, $b$ is a bias term, $f$ is the activation function, and $y$ is the output. The weights control the strength of the connections between neurons, while the activation function introduces a non-linearity that allows the network to model complex patterns.

Without activation functions, neural networks would be limited to learning linear combinations of their inputs. No matter how many layers you stack, the end-to-end network would still be computing a linear function. In contrast, non-linear activation functions expand the range of possible functions a neural network can model to include curves, step functions, and other complex shapes.

The choice of activation function is a key hyperparameter when designing a neural network architecture. It influences the network‘s ability to model different function classes, the speed and stability of the learning process, the sparsity of learned representations, and more. Let‘s tour the most common activation functions in use today and examine their properties.

Sigmoid Activations

Sigmoids were the earliest and most widely used activation functions in neural networks. The standard logistic sigmoid is defined as:

$$
\sigma(z) = \frac{1}{1+e^{-z}}
$$

It maps inputs to an S-shaped curve between 0 and 1:

Sigmoids have several properties that made them historically useful:

• They are bounded between 0 and 1, providing a probabilistic interpretation of a neuron‘s output
• They are differentiable, enabling gradient-based learning
• They exhibit a degree of biological plausibility in modeling the firing rates of neurons

However, sigmoids have fallen out of favor in recent years due to some major drawbacks:

• The vanishing gradient problem: When inputs become large (negative or positive), the sigmoid saturates at 0 or 1 and the gradient becomes nearly zero. This slows learning to a crawl.
• Outputs are not zero-centered: This causes problems for subsequent layers and slows convergence.
• Saturated neurons "die": Neurons with near-zero gradient stop contributing and drop out of the learning process.
• Computationally expensive: The exponential function is slower to calculate than alternatives.

Today, sigmoids are primarily used in the output layer for binary classification problems. The softmax function (a generalization of the sigmoid) remains popular for multi-class problems.

Hyperbolic Tangent (tanh)

The tanh function is very similar to the logistic sigmoid but maps inputs to a range of -1 to 1:

$$
tanh(z) = \frac{e^z – e^{-z}}{e^z + e^{-z}}
$$

The output of tanh is zero-centered which helps alleviate some of the convergence issues with sigmoids. However, tanh still suffers from vanishing gradients and saturation at the tails. Like sigmoids, tanh activations have been largely replaced by ReLU variants in modern networks.

Rectified Linear Unit (ReLU)

The ReLU has emerged as the default activation function used in most feed-forward neural networks. It simply computes the function:

$$
f(z) = max(0, z)
$$

Resulting in a piecewise linear function:

ReLUs have several major advantages over sigmoids and tanh:

• Sparse representations: Since ReLUs output 0 for negative inputs, they naturally lead to sparse representations with many neurons inactive.
• No vanishing gradient: The derivative of ReLUs is 1 for positive inputs, so gradients flow through the network more easily.
• Computationally efficient: Only a simple max() operation is required.

These properties allow ReLU networks to converge much faster–up to 6x faster than tanh networks. ReLU variants like leaky and parametric ReLUs aim to address the main weakness of ReLUs known as the "dying ReLU" problem. Since the gradient is 0 for negative inputs, some neurons may permanently die and stop learning if they get stuck in the flat negative region.

ReLUs remain the most popular default choice of activation across many domains. However, a promising challenger known as SELU will be discussed next.

Scaled Exponential Linear Units (SELUs)

The scaled exponential linear unit (SELU) was proposed in 2017 as an activation to enable self-normalizing neural networks that are highly robust to perturbations in inputs and weights. The SELU function is defined as:

$$
f(z) = \lambda
\begin{cases}
z, & \text{if } z > 0\
\alpha e^z – \alpha, & \text{if } z \leq 0
\end{cases}
$$

where $\lambda=1.0507$ and $\alpha=1.6733$ are pre-defined constants. The graph of SELU looks like:

SELU was derived by solving for an activation that maps the mean and variance of layer inputs to a fixed point of 0 mean and unit variance. This property causes SELU networks to self-normalize–the distributions of layer activations automatically converge towards zero mean and unit variance as the network depth increases.

Self-normalization brings several benefits:

• Gradients are normalized which avoids vanishing or exploding problems
• No batch normalization is required, reducing complexity and computation
• Networks become highly robust to weight initialization and perturbations
• Learning is less sensitive to hyperparameters like learning rate and regularization

Empirically, SELU networks have demonstrated state-of-the-art results on several challenging image classification benchmarks. The theory of SELU is an active area of research that may lead to even more powerful self-normalizing architectures.

The main downside of SELU is that it requires careful architecture design and has some overhead due to the exponential function. But for deep feed-forward networks, SELU is becoming an increasingly attractive option.

Comparing Activation Functions

The following table summarizes the key properties of the activation functions covered:

Activation	Range	Pros	Cons	Use Cases
Sigmoid	(0, 1)	Differentiable, Biologically plausible	Vanishing gradient, Not zero-centered, Saturates, Slow	Binary outputs
tanh	(-1, 1)	Zero-centered	Vanishing gradient, Saturates, Slow	Rarely used today
ReLU	[0, ∞)	Sparse outputs, Efficient, No saturation	Dying ReLU problem	Most feed-forward nets
SELU	(-λα, ∞)	Self-normalizing, Robust, Efficient	Requires careful design	Deep feed-forward nets

While other activation functions like softmax, ELUs, and swish are also used, the above are the most common. Visualization can be a helpful tool to understand their differing properties.

The following plot shows how a toy 3-layer neural network (50 neurons per layer) with different activations maps inputs from a 2D coordinate grid to an output space:

Image Credit: playground.tensorflow.org

The sigmoid network learns a relatively smooth mapping, while ReLU and SELU produce more angular mappings with sharper transitions. This reflects the ability of ReLU-based nets to model more complex functions given the same architecture.

Guidance on Selecting Activations

With numerous activation functions to choose from, it can be daunting to decide which one to use. While there‘s no one-size-fits-all answer, here are some rules of thumb grounded in the latest research and practical experience:

• Use ReLUs as the default starting point unless you have a specific reason to use another activation.
• For output layers, let the problem type guide your choice: sigmoid for binary outputs, softmax for multi-class, and linear for regression.
• If hitting a performance wall with ReLUs, experiment with SELU, ELUs or leaky ReLUs, especially in deeper networks.
• Be mindful of normalization–ReLUs often require batch norm for effective training. SELUs have normalization built-in.
• Residual and skip connections can help alleviate the vanishing gradient problem in very deep sigmoid/tanh networks.
• Monitor the distributions of layer activations during training–they should be stable and not diverge.
• Glorot or He initialization are good default schemes that are derived to work well with different activations.
• When in doubt, try a few different activations and let empirical results guide your choice!

Proper activation function selection goes hand-in-hand with architectural choices (depth, width, layer type), initialization, normalization, and optimization. It‘s an important knob to tune as part of the overall deep learning pipeline.

Several large-scale studies have compared activations across a variety of models and datasets. One study found that while ReLUs remain a solid default choice, ELUs and SELUs outperform on certain challenging computer vision tasks. Another found that learned parametric activations tuned for a given architecture can boost performance.

Novel activations are an active area of research–the Gaussian Error Linear Unit (GELU), Mish, and self-attention based activations show promising results on language and vision benchmarks. New activation functions are typically simple to implement and test, so the field evolves quickly.

Biological Basis and Future Directions

The human brain has been a key source of inspiration for the design of artificial neural networks. It‘s interesting to consider how biological neurons compute their activations and what lessons we can draw for ANNs.

Biological neurons communicate via electrical impulses known as action potentials or "spikes". The spike rate of a neuron depends on the input current from connected neurons. This spike rate is often modeled with an exponential sigmoid-like function. However, real neurons exhibit far richer and more complex temporal dynamics than ANNs.

Several promising research directions aim to bridge this gap between artificial and biological neural networks:

• Spiking neural networks that mimic the temporal dynamics of biological neurons
• Neural networks with more complex dendritic computation and branch-specific plasticity
• Architectures inspired by the layered structure and inter-region wiring of the cortex
• Activations with attentional gating and other higher-order interactions
• Incorporation of top-down feedback and recurrence akin to biological circuits

As we deepen our understanding of both biological and artificial neural networks, we can expect continued cross-pollination of ideas. More computationally efficient and biologically plausible activations may not only improve AI systems but also advance our knowledge of natural intelligence.

Conclusion

Activation functions are a key ingredient in the success of modern deep learning. They introduce the non-linearity that allows neural networks to model highly complex functions and learn rich representations. Understanding the properties, strengths and weaknesses of different activation functions is essential for designing effective architectures.

While classic choices like sigmoids and ReLUs remain popular, novel activations like SELU are pushing performance to new heights on certain problems. The field continues to evolve quickly, with biological inspiration, theoretical analysis, and empirical studies lighting the way.

Ultimately, the choice of activation function should be guided by a mix of domain knowledge, empirical results and fundamental understanding. By mastering these powerful functions, deep learning practitioners can design more efficient, robust and capable models to tackle diverse challenges. As an expanding universe of activation functions meets ever-richer neural network architectures, the future is bright for unlocking the full potential of deep learning.