Which activation function is commonly used in the hidden layers of a neural network to introduce non-linearity?

The Rectified Linear Unit (ReLU) activation function is commonly used in the hidden layers of a neural network to introduce non-linearity.

Here are the steps to understand why:

1. Non-linearity: In a neural network, we need the activation function to introduce non-linearity into the network. Without it, no matter how many layers we have, the neural network would behave just like a single layer.

2. ReLU Function: The ReLU function is defined as the positive part of its argument. It outputs the input directly if it is positive, otherwise, it will output zero. It has become the default activation function for many types of neural networks because a model that uses it is easier to train and often achieves better performance.

3. Advantages of ReLU: The advantages of using ReLU as an activation function are that it does not activate all the neurons at the same time. This means that the neurons will only be deactivated if the output of the linear transformation is less than 0. This makes ReLU networks computationally efficient and easy to compute, as sparse activation is more biologically plausible.

4. Other Activation Functions: Other activation functions like sigmoid or hyperbolic tangent also introduce non-linearity. However, they are less used in practice because they suffer from the vanishing gradient problem, which slows down the learning.

So, to summarize, the ReLU activation function is commonly used in the hidden layers of a neural network to introduce non-linearity due to its computational efficiency and performance.