What Is the Sigmoid Function? How It Works in Neural Networks Explained

Sigmoid is a mathematical function that maps any real-valued number into a value between 0 and 1. Its characteristic "S"-shaped curve makes it particularly useful in scenarios where we need to convert outputs into probabilities. This function is often called the logistic function.

Mathematically, sigmoid is represented as:

Sigmoid function is used as an activation function in machine learning and neural networks for modelling binary classification problems, smoothing outputs, and introducing non-linearity into models.

In this graph, the x-axis represents the input values that ranges from −∞ to +∞−∞ to +∞ and y-axis represents the output values which always lie in [0,1].

In machine learning, x could be a weighted sum of inputs in a neural network neuron or a raw score in logistic regression. If the output value is close to 1, it indicates high confidence in one class and if the value is close to 0, it indicates high confidence in the other class. 

The sigmoid function has several key properties that make it a popular choice in machine learning and neural networks:

1. Domain: The domain of the sigmoid function is all real numbers. This means that you can input any real number into the sigmoid function, and it will produce a valid output.
2. Asymptotes: As xx approaches positive infinity, σ(x)σ(x) approaches 1. Conversely, as xx approaches negative infinity, σ(x)σ(x) approaches 0. This property ensures that the function never actually reaches 0 or 1, but gets arbitrarily close.
3. Monotonicity: The sigmoid function is monotonically increasing, meaning that as the input increases, the output also increases.
4. Differentiability: The sigmoid function is differentiable, which allows for the calculation of gradients during the training of machine learning models.

Sigmoid Function in Backpropagation

If we use a linear activation function in a neural network, the model will only be able to separate data linearly, which results in poor performance on non-linear datasets. However, by adding a hidden layer with a sigmoid activation function, the model gains the ability to handle non-linearity, thereby improving performance.

During the backpropagation, the model calculates and updates weights and biases by computing the derivative of the activation function. The sigmoid function useful because:

• It is the only function that appears in its derivative
• It is differentiable at every point, which helps in the effective computation of gradients during backpropagation

The derivative of the sigmoid function, denoted as σ′(x)σ′(x), is given by σ′(x)=σ(x)⋅(1−σ(x))σ′(x)=σ(x)⋅(1−σ(x)).

The below image shows the derivative of the sigmoid function graphically.

Issue with Sigmoid Function in Backpropagation

One key issue with using the sigmoid function is the vanishing gradient problem. When updating weights and biases using gradient descent, if the gradients are too small, the updates to weights and biases become insignificant, slowing down or even stopping learning.

The shades red region highlights the areas where the derivative σ′(x)σ′(x) is very small (close to 0). In these regions, the gradients used to update weights and biases during backpropagation become extremely small. As a result, the model learns very slowly or stops learning altogether, which is a major issue in deep neural networks.