Activation Functions Tutorial: Deep Learning with Tensorflow, Keras, and Python

In deep learning, activation functions play a crucial role in determining the output of a neural network. They introduce non-linearity into the network, allowing it to learn complex patterns in the data. In this tutorial, we will discuss some of the commonly used activation functions in deep learning, such as sigmoid, tanh, ReLU, and softmax. We will also implement these activation functions using TensorFlow and Keras in Python.

1. Sigmoid Activation Function:

The sigmoid activation function is a commonly used activation function in neural networks. It takes an input value and outputs a value between 0 and 1, which can be interpreted as the probability of the neuron being activated. The sigmoid function is defined as:

S(x) = 1 / (1 + e^(-x))

To implement the sigmoid activation function in TensorFlow, you can use the tf.keras.activations.sigmoid() function. Here is an example code snippet:

import tensorflow as tf
import numpy as np

# Input data
x = np.array([-1, 0, 1, 2, 3], dtype=np.float32)

# Define the sigmoid activation function
def sigmoid(x):
    return 1 / (1 + tf.exp(-x))

# Calculate the output
output = sigmoid(x)
print(output)

1. Tanh Activation Function:

The tanh activation function is another commonly used activation function in neural networks. It takes an input value and outputs a value between -1 and 1. The tanh function is defined as:

tanh(x) = (e^x – e^(-x)) / (e^x + e^(-x))

To implement the tanh activation function in TensorFlow, you can use the tf.keras.activations.tanh() function. Here is an example code snippet:

# Define the tanh activation function
def tanh(x):
    return tf.math.tanh(x)

# Calculate the output
output = tanh(x)
print(output)

1. ReLU Activation Function:

The ReLU (Rectified Linear Unit) activation function is a widely used activation function in deep learning. It replaces all negative values in the input with zero, while leaving positive values unchanged. The ReLU function is defined as:

ReLU(x) = max(0, x)

To implement the ReLU activation function in TensorFlow, you can use the tf.keras.activations.relu() function. Here is an example code snippet:

# Define the ReLU activation function
def relu(x):
    return tf.maximum(0, x)

# Calculate the output
output = relu(x)
print(output)

1. Softmax Activation Function:

The softmax activation function is often used in the output layer of a neural network for multi-class classification problems. It takes an input vector and normalizes it into a probability distribution over all classes. The softmax function is defined as:

softmax(x_i) = e^(x_i) / sum(e^(x_j))

To implement the softmax activation function in TensorFlow, you can use the tf.keras.activations.softmax() function. Here is an example code snippet:

# Define the softmax activation function
def softmax(x):
    return tf.nn.softmax(x)

# Calculate the output
output = softmax(x)
print(output)

In this tutorial, we discussed some commonly used activation functions in deep learning and implemented them using TensorFlow and Keras in Python. Activation functions play a crucial role in the performance of neural networks, so it is important to choose the right activation function based on the problem at hand. I hope this tutorial was helpful in understanding activation functions in deep learning.