Weight Initialisation techniques in Pytorch

In neural networks, weight initialisation is an important process that can impact the performance of the model. Proper weight initialisation can help improve the training process and prevent issues like vanishing or exploding gradients.

Quick Walkthrough

PyTorch provides several built-in weight initialisation techniques that you can use in your models. Some common techniques include:

• Normal weight initialisation
• Xavier weight initialisation
• Kaiming weight initialisation

Tutorial for Beginners

If you’re new to PyTorch and want to learn how to use weight initialisation techniques, here’s a quick tutorial:

1. Import the necessary libraries:


import torch
import torch.nn as nn
import torch.nn.init as init



2. Define your neural network architecture:


class MyModel(nn.Module):
def __init__(self):
super(MyModel, self).__init__()
self.fc1 = nn.Linear(784, 128)
self.fc2 = nn.Linear(128, 10)



3. Apply weight initialisation:


def init_weights(m):
if type(m) == nn.Linear:
init.xavier_uniform(m.weight)


model = MyModel()
model.apply(init_weights)


By following these steps, you can easily apply weight initialisation techniques in your PyTorch models and improve their performance.