How to Use TensorFlow Serving for Deep Learning – Tutorial 48 (TensorFlow, Python)

In this tutorial, we are going to cover TensorFlow Serving, which is a system for serving machine learning models in production environments. TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments. It has a flexible architecture that allows you to deploy new algorithms and experiment with them easily.

For this tutorial, we are going to assume that you have some basic knowledge of TensorFlow and Python. If you are new to TensorFlow, we recommend checking out some of the introductory tutorials available on the TensorFlow website.

Let’s get started with our tutorial on TensorFlow Serving!

Step 1: Installing TensorFlow Serving

The first step is to install TensorFlow Serving on your machine. You can install TensorFlow Serving using Docker, but for this tutorial, we are going to install it using pip. Open your terminal and run the following command:

pip install tensorflow-serving-api

Step 2: Building a TensorFlow model

Next, we need to build a TensorFlow model that we want to serve using TensorFlow Serving. For this tutorial, we are going to build a simple model that classifies images of handwritten digits from the MNIST dataset. Here is the code to build the model:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Flatten, Dense

# Load the MNIST dataset
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Normalize the pixel values
x_train, x_test = x_train / 255.0, x_test / 255.0

# Build the model
model = Sequential([
    Flatten(input_shape=(28, 28)),
    Dense(128, activation='relu'),
    Dense(10, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Train the model
model.fit(x_train, y_train, epochs=5, validation_data=(x_test, y_test))

Step 3: Exporting the model for TensorFlow Serving

Once we have trained our model, we need to export it in a format that TensorFlow Serving can understand. We can do this using the SavedModel format. Here is the code to export the model:

import os

# Define the directory to save the model
export_path = 'saved_model/1'

# Save the model in the SavedModel format
model.save(export_path)

Step 4: Running TensorFlow Serving

Now that we have exported our model, we can start TensorFlow Serving to serve it. Open your terminal and run the following command:

tensorflow_model_server --port=8500 --rest_api_port=8501 --model_name=mnist --model_base_path=/path/to/exported_model

Make sure to replace /path/to/exported_model with the path to the directory where you saved your exported model.

Step 5: Making predictions using the served model

Once TensorFlow Serving is running, we can make predictions using the served model. We can use the REST API provided by TensorFlow Serving to make predictions. Here is an example code to make predictions using the served model:

import requests
import json

# Define the input data
data = json.dumps({"signature_name": "serving_default", "instances": x_test[0:3].tolist()})

# Make a POST request to the prediction endpoint
headers = {"content-type": "application/json"}
response = requests.post('http://localhost:8501/v1/models/mnist:predict', data=data, headers=headers)

# Print the predictions
predictions = json.loads(response.text)['predictions']
print(predictions)

And that’s it! You have successfully served a TensorFlow model using TensorFlow Serving and made predictions using the served model.

In this tutorial, we covered how to install TensorFlow Serving, build a TensorFlow model, export the model for serving, run TensorFlow Serving, and make predictions using the served model. TensorFlow Serving is a powerful tool for serving machine learning models in production environments, and we hope this tutorial helped you understand how to use it.

If you have any questions or run into any issues, feel free to ask for help in the comments section. Happy coding!