Mastering Activation Functions in PyTorch: A Deep Dive Tutorial
In this tutorial, we will explore the different types of activation functions in PyTorch and how to effectively use them in your deep learning models. Activation functions play a crucial role in neural networks by introducing non-linearity to the model. They help in determining whether a neuron should be activated or not based on the input it receives.
We will cover the following activation functions in PyTorch:
- Sigmoid
- Tanh
- ReLU
- Leaky ReLU
- PReLU
- ELU
- SELU
- Softmax
Let’s start by importing the necessary libraries and setting up our environment.
<!DOCTYPE html>
<html>
<head>
<title>Mastering Activation Functions in PyTorch</title>
<script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/torch/1.9.0/torch.min.js"></script>
</head>
<body>
<h1>Mastering Activation Functions in PyTorch</h1>
<p>We will cover the different types of activation functions in PyTorch and how to use them effectively.</p>
<script>
// Your PyTorch code goes here
</script>
</body>
</html>Now, let’s delve into each activation function and see how they can be implemented in PyTorch.
1. Sigmoid
The sigmoid function is defined as:
sigmoid(x) = 1 / (1 + e^(-x))It squashes the output to a range between 0 and 1, which is useful for binary classification problems.
<script>
const x = torch.tensor([1.0, 2.0, 3.0]);
const sigmoid = torch.sigmoid(x);
console.log(sigmoid);
</script>2. Tanh
The hyperbolic tangent (tanh) function is defined as:
tanh(x) = (e^x - e^(-x)) / (e^x + e^(-x))It squashes the output to a range between -1 and 1, which helps in convergence during training.
<script>
const x = torch.tensor([1.0, 2.0, 3.0]);
const tanh = torch.tanh(x);
console.log(tanh);
</script>3. ReLU
The Rectified Linear Unit (ReLU) function is defined as:
ReLU(x) = max(0, x)It introduces non-linearity by setting negative values to zero.
<script>
const x = torch.tensor([1.0, -2.0, 3.0]);
const relu = torch.relu(x);
console.log(relu);
</script>4. Leaky ReLU
The Leaky ReLU function is defined as:
LeakyReLU(x) = max(0.01x, x)It addresses the dying ReLU problem by allowing a small gradient for negative values.
<script>
const x = torch.tensor([1.0, -2.0, 3.0]);
const leaky_relu = torch.leaky_relu(x, 0.01);
console.log(leaky_relu);
</script>5. PReLU
The Parametric ReLU (PReLU) function is defined as:
PReLU(x) = max(alpha * x, x)It allows the learnable parameter alpha to be adjusted during training.
<script>
const x = torch.tensor([1.0, -2.0, 3.0]);
const alpha = torch.tensor([0.1, 0.2, 0.3]);
const prelu = torch.nn.functional.prelu(x, alpha);
console.log(prelu);
</script>6. ELU
The Exponential Linear Unit (ELU) function is defined as:
ELU(x) = x if x > 0, else alpha * (e^x - 1)It smooths the ReLU function by allowing negative values with a smooth gradient.
<script>
const x = torch.tensor([1.0, -2.0, 3.0]);
const elu = torch.nn.functional.elu(x);
console.log(elu);
</script>7. SELU
The Scaled Exponential Linear Unit (SELU) function is defined as:
SELU(x) = scale * (x if x > 0, else alpha * (e^x - 1))It is a self-normalizing activation function that improves training stability.
<script>
const x = torch.tensor([1.0, -2.0, 3.0]);
const selu = torch.nn.functional.selu(x);
console.log(selu);
</script>8. Softmax
The Softmax function is defined as:
softmax(x_i) = e^(x_i) / sum(e^(x))It is used for multi-class classification problems to normalize the outputs into a probability distribution.
<script>
const x = torch.tensor([1.0, 2.0, 3.0]);
const softmax = torch.nn.functional.softmax(x, -1);
console.log(softmax);
</script>Congratulations! You have now mastered different activation functions in PyTorch. Experiment with these functions in your neural network models to achieve better performance.
Thanks for checking out this video! I'm going to post a few important links down below
PyTorch Playlist: https://www.youtube.com/playlist?list=PLcQVY5V2UY4KzVIok0mWdp-zigfdOKZI-
Twitter: https://twitter.com/RyanNolanData
LinkedIn: https://www.linkedin.com/in/ryan-p-nolan/