Recognizing Handwritten Digits in Real-Time using PyTorch Neural Networks in Houdini

Real-Time Handwritten Digit Recognition in Houdini with PyTorch Neural Networks

In this tutorial, we will explore how to create a real-time handwritten digit recognition system using PyTorch neural networks in Houdini. Handwritten digit recognition is a common task in machine learning and computer vision, and by using PyTorch, we can easily create and train deep learning models to recognize digits in real-time.

Step 1: Setting up the Environment
Before we start building the handwriting recognition system in Houdini, we need to set up the environment by installing PyTorch and any necessary dependencies. Make sure you have Python installed on your system, as PyTorch is a Python library.

To install PyTorch, you can use pip by running the following command:

pip install torch torchvision

Step 2: Creating the Neural Network Model
To recognize handwritten digits, we need to train a neural network model using PyTorch. In this tutorial, we will use a simple convolutional neural network (CNN) model for digit recognition.

Here is an example of a simple CNN model using PyTorch:

import torch.nn as nn
import torch.nn.functional as F

class DigitRecognitionCNN(nn.Module):
    def __init__(self):
        super(DigitRecognitionCNN, self).__init__()

        self.conv1 = nn.Conv2d(1, 32, kernel_size=5)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=5)
        self.fc1 = nn.Linear(1024, 128)
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2)
        x = x.view(-1, 1024)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

This CNN model takes a 28×28 image of a handwritten digit and outputs a log probability distribution over the digits 0-9.

Step 3: Building the Real-Time Handwritten Digit Recognition System in Houdini
To build the real-time handwritten digit recognition system in Houdini, we will use the Houdini Python SOP node to capture and process the input image of a handwritten digit, and then use the trained neural network model to recognize the digit.

Here is the HTML code for the Houdini Python SOP node to implement the real-time handwritten digit recognition system:

<pythonSOP>
import torch
from PIL import Image
from torchvision.transforms import ToTensor
from DigitRecognitionCNN import DigitRecognitionCNN

class DigitRecognitionSOPNode:
    def __init__(self):
        self.model = DigitRecognitionCNN()
        self.model.load_state_dict(torch.load('digit_recognition_model.pth'))
        self.model.eval()

    def cook(self, geometry):
        for point in geometry.points():
            # Get the x, y coordinates of the point
            x, y = point.position()

            # Get the color value of the pixel at the x, y coordinates
            pixel_value = hou.Color((x, y, 0))

            # Convert the pixel value to a grayscale image
            image = Image.new('L', (28, 28))
            for i in range(28):
                for j in range(28):
                    image.putpixel((i, j), int(pixel_value[i][j] * 255))

            # Preprocess the image for input to the neural network model
            image = ToTensor()(image).unsqueeze(0)

            # Make a prediction using the neural network model
            with torch.no_grad():
                output = self.model(image)

            # Get the predicted digit from the output
            prediction = torch.argmax(output, dim=1).item()

            # Set the point attribute to the predicted digit
            point.setAttribValue('digit', prediction)

DigitRecognitionSOPNode().cook(geometry)
</pythonSOP>

This Python SOP node captures the input image of a handwritten digit as a grayscale image, preprocesses the image for input to the neural network model, makes a prediction using the trained neural network model, and sets the point attribute to the predicted digit.

Step 4: Training the Neural Network Model
Before using the real-time handwritten digit recognition system in Houdini, we need to train the convolutional neural network model using a dataset of handwritten digits, such as the MNIST dataset.

To train the neural network model, you can use the following Python code:

import torch
import torch.optim as optim
from torchvision.datasets import MNIST
from torchvision.transforms import ToTensor
from torch.utils.data import DataLoader
from DigitRecognitionCNN import DigitRecognitionCNN

# Load the MNIST dataset
train_dataset = MNIST(root='./data', train=True, transform=ToTensor(), download=True)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)

# Initialize the neural network model
model = DigitRecognitionCNN()

# Define the loss function and optimizer
criterion = torch.nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Train the neural network model
for epoch in range(10):
    for i, (images, labels) in enumerate(train_loader):
        optimizer.zero_grad()
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

# Save the trained model
torch.save(model.state_dict(), 'digit_recognition_model.pth')

This Python code loads the MNIST dataset, initializes the CNN model, defines the loss function and optimizer, trains the model using the dataset, and saves the trained model to a file.

Step 5: Implementing the Real-Time Handwritten Digit Recognition System in Houdini
To implement the real-time handwritten digit recognition system in Houdini, follow these steps:

1. Create a new Python SOP node in Houdini.
2. Copy and paste the HTML code for the Python SOP node provided in Step 3.
3. Save the Python SOP node and set the Python SOP node’s Python script parameter to point to the saved Python script.
4. Connect the Python SOP node to a geometry node in the Houdini network editor.
5. Add a camera node to the network editor to capture the input image of a handwritten digit.
6. Add a point attribute called "digit" to the geometry node to store the predicted digit.

With the real-time handwritten digit recognition system implemented in Houdini, you can now draw or write digits in the Houdini viewport, and the neural network model will recognize and display the predicted digit in real-time using PyTorch neural networks.

In conclusion, this tutorial has demonstrated how to create a real-time handwritten digit recognition system in Houdini using PyTorch neural networks. By following the steps outlined in this tutorial, you can build your own real-time handwriting recognition system and explore other applications of deep learning models in Houdini.