Search

# Sigmoid Neuron — Building Block of Deep Neural Networks

The building block of the deep neural networks is called the sigmoid neuron. Sigmoid neurons are similar to perceptrons, but they are slightly modified such that the output from the sigmoid neuron is much smoother than the step functional output from perceptron. In this post, we will talk about the motivation behind the creation of sigmoid neuron and working of the sigmoid neuron model.

This is the 1st part in the two-part series discussing the working of sigmoid neuron and it’s learning algorithm:

1 | Sigmoid Neuron — Building Block of Deep Neural Networks

Why Sigmoid Neuron?

Before we go into the working of a sigmoid neuron, let's talk about the perceptron model and its limitations in brief.

Perceptron model takes several real-valued inputs and gives a single binary output. In the perceptron model, every input xi has weight wi associated with it. The weights indicate the importance of the input in the decision-making process. The model output is decided by a threshold Wₒ if the weighted sum of the inputs is greater than threshold Wₒ output will be 1 else output will be 0. In other words, the model will fire if the weighted sum is greater than the threshold.

Perceptron (Left) & Mathematical Representation (Right)From the mathematical representation, we might say that the thresholding logic used by the perceptron is very harsh. Let’s see the harsh thresholding logic with an example. Consider the decision making process of a person, whether he/she would like to purchase a car or not based on only one input X1 — Salary and by setting the threshold b(Wₒ) = -10 and the weight W = 0.2. The output from the perceptron model will look like in the figure shown below.Data (Left) & Graphical Representation of Output(Right) 📷 📷 Red points indicates that a person would not buy a car and green points indicate that person would like to buy a car. Isn’t it a bit odd that a person with 50.1K will buy a car but someone with a 49.9K will not buy a car? The small change in the input to a perceptron can sometimes cause the output to completely flip, say from 0 to 1. This behavior is not a characteristic of the specific problem we choose or the specific weight and the threshold we choose. It is a characteristic of the perceptron neuron itself which behaves like a step function. We can overcome this problem by introducing a new type of artificial neuron called a sigmoid neuron. To know more about the working of the perceptron, kindly refer to my previous post on the Perceptron Model