Activation Functions in Neural Networks: Types, Uses, and How They Work

An activation function is applied to the weighted sum of inputs (before producing the final output of a neuron). It introduces non-linearity, enabling the model to learn and represent complex data patterns. Without it, even a deep neural network would behave like a simple linear regression model.

Figure 1. Activation function in Neural Networks

Activation functions decide whether a neuron should be activated based on the weighted sum of inputs and a bias term. They also make backpropagation possible by providing gradients for weight updates.

Why Non-Linearity is Important

Real-world data is rarely linearly separable.
Non-linear functions allow neural networks to form curved decision boundaries, making them capable of handling complex patterns (e.g., classifying apples vs. bananas under varying colors and shapes).
They ensure networks can model advanced problems like image recognition, NLP and speech processing.

Mathematical Example

Consider a neural network with:

Inputs: i1, i2
Hidden layer: neurons h1 and h2
Output layer: one neuron (output)
Weights: w1, w2, w3, w4, w5, w6
Biases: b1 for hidden layer, b2 for output layer

Figure 2. Demonstration of neural network

Types of Activation Functions in Deep Learning

1. Linear Activation Function

Linear Activation Function resembles straight line define by y=x. No matter how many layers the neural network contains if they all use linear activation functions the output is a linear combination of the input.

The range of the output spans from (−∞ to +∞)(−∞ to +∞).
Linear activation function is used at just one place i.e. output layer.
Using linear activation across all layers makes the network's ability to learn complex patterns limited.

Linear activation functions are useful for specific tasks but must be combined with non-linear functions to enhance the neural network’s learning and predictive capabilities.

2. Non-Linear Activation Functions

1. Sigmoid Function

Sigmoid Activation Function is characterized by ‘s’ shape. This formula ensures a smooth and continuous output that is essential for gradient-based optimization methods.

It allows neural networks to handle and model complex patterns that linear equations cannot.
The output ranges between 0 and 1, hence useful for binary classification.
The function exhibits a steep gradient when x values are between -2 and 2. This sensitivity means that small changes in input x can cause significant changes in output y which is critical during the training process.

2. Tanh Activation Function

Tanh function (hyperbolic tangent function) is a shifted version of the sigmoid, allowing it to stretch across the y-axis. It is defined as:

Figure 5. Tanh Activation Function formula

Alternatively, it can be expressed using the sigmoid function: tanh(x)=2×sigmoid(2x)−1

Value Range: Outputs values from -1 to +1.
Non-linear: Enables modeling of complex data patterns.
Use in Hidden Layers: Commonly used in hidden layers due to its zero-centered output, facilitating easier learning for subsequent layers.

3. ReLU (Rectified Linear Unit) Function

ReLU activation is defined by A(x)=max⁡(0,x), this means that if the input x is positive, ReLU returns x, if the input is negative, it returns 0.

Value Range: [0,∞)[0,∞), meaning the function only outputs non-negative values.
Nature: It is a non-linear activation function, allowing neural networks to learn complex patterns and making backpropagation more efficient.
Advantage over other Activation: ReLU is less computationally expensive than tanh and sigmoid because it involves simpler mathematical operations. At a time only a few neurons are activated making the network sparse making it efficient and easy for computation.

4. Leaky ReLU

Leaky ReLU is similar to ReLU but allows a small negative slope (αα, e.g., 0.01) instead of zero.
Solves the “dying ReLU” problem, where neurons get stuck with zero outputs.
Range: (−∞,∞)(−∞,∞).
Preferred in some cases for better gradient flow.

Figure 9. Leaky ReLU Activation Function

5. SoftPlus Function

Softplus function is defined mathematically as: A(x)=log(1+ex).

This equation ensures that the output is always positive and differentiable at all points which is an advantage over the traditional ReLU function.

Nature: The Softplus function is non-linear.
Range: The function outputs values in the range (0,∞)(0,∞), similar to ReLU, but without the hard zero threshold that ReLU has.
Smoothness: Softplus is a smooth, continuous function, meaning it avoids the sharp discontinuities of ReLU which can sometimes lead to problems during optimization.

3. Exponential Linear Units

1. ELU (Exponential Linear Unit) Function

ELU (Exponential Linear Unit) is a non-linear activation function designed to improve learning speed and reduce the vanishing gradient problem. It behaves like ReLU for positive inputs but allows smooth negative values instead of zero, which helps the network learn more balanced representations.

Figure 11. ELU activation function formula

Value Range: (−α,∞)(-\alpha, \infty)(−α,∞)
Non-linear: Helps neural networks model complex patterns
Negative Outputs: Unlike ReLU, ELU allows negative values, making outputs closer to zero mean
Smooth Curve: Continuous and differentiable, helping stable training

Figure 12. ELU Activation function graph

2. SELU (Scaled Exponential Linear Unit) Function

SELU is a scaled version of ELU designed for self-normalizing neural networks. It automatically keeps neuron outputs close to zero mean and unit variance, which stabilizes training. It is defined as:

Value Range: (−λα,∞)(−λα,∞)
Self-Normalizing: Keeps activations stable across layers.
Improves Gradient Flow: Helps prevent exploding or vanishing gradients.
Works Best With: Deep fully connected networks using proper initialization and dropout variants.
Reduces Need for Batch Normalization in some architectures.

4. Output Layer Activation Functions

1. Sigmoid Activation Function

Sigmoid function produces an S-shaped curve and maps input values into a probability-like range between 0 and 1 and is used to find the final output of the neural network for binary classification problems.

Value Range: (0,1)(0,1)(0,1)
Smooth and Differentiable: Useful for gradient-based optimization.
Best for: Binary classification output layers.
Interpretable Output: Can be treated as probability.
Limitation: Suffers from vanishing gradient for very large or very small inputs.

Figure 15. Sigmoid function activation graph

2. Softmax Function

Softmax function is designed to handle multi-class classification problems. It transforms raw output scores from a neural network into probabilities. It works by squashing the output values of each class into the range of 0 to 1 while ensuring that the sum of all probabilities equals 1.

Softmax is a non-linear activation function.
The Softmax function ensures that each class is assigned a probability, helping to identify which class the input belongs to.

Impact of Activation Functions on Model Performance

The choice of activation function has a direct impact on the performance of a neural network in several ways:

Convergence Speed: Functions like ReLU allow faster training by avoiding the vanishing gradient problem while Sigmoid and Tanh can slow down convergence in deep networks.
Gradient Flow: Activation functions like ReLU ensure better gradient flow, helping deeper layers learn effectively. In contrast Sigmoid can lead to small gradients, hindering learning in deep layers.
Model Complexity: Activation functions like Softmax allow the model to handle complex multi-class problems, whereas simpler functions like ReLU or Leaky ReLU are used for basic layers.