Building a Producer and Consumer using Apache Kafka and Node.js

Apache Kafka is a distributed streaming platform that allows you to publish and subscribe to streams of records. In this tutorial, we will learn how to create a producer and consumer in Node.js using the kafka-node library.

Step 1: Setting up Apache Kafka
Before you start creating your Node.js producer and consumer, you need to set up Apache Kafka on your local machine or on a remote server. You can follow the official Apache Kafka documentation to set up Apache Kafka on your machine.

Step 2: Install kafka-node library
You will need to install the kafka-node library in your Node.js project to interact with Apache Kafka. You can install the kafka-node library using npm with the following command:

npm install kafka-node

Step 3: Creating a Kafka Producer
Now, let’s create a Kafka producer in Node.js. In a new Node.js file, import the kafka-node library and create a new Kafka producer.

const kafka = require('kafka-node');
const Producer = kafka.Producer;
const client = new kafka.KafkaClient();

const producer = new Producer(client);

Next, we need to connect the producer to the Kafka broker and send messages to a Kafka topic. We can use the send method of the producer to send messages to a topic.

producer.on('ready', () => {
  const payloads = [
    { topic: 'my-topic', messages: 'Hello, Kafka!' }
  ];

  producer.send(payloads, (err, data) => {
    if (err) {
      console.error(err);
    } else {
      console.log(data);
    }
  });
});

Step 4: Creating a Kafka Consumer
Now, let’s create a Kafka consumer in Node.js. In a new Node.js file, import the kafka-node library and create a new Kafka consumer.

const kafka = require('kafka-node');
const Consumer = kafka.Consumer;
const client = new kafka.KafkaClient();

const consumer = new Consumer(client, [{ topic: 'my-topic' }]);

Next, we need to listen for messages from the Kafka topic. We can use the on('message') method of the consumer to listen for messages.

consumer.on('message', (message) => {
  console.log(message);
});

Step 5: Running the Producer and Consumer
Now that we have created our Kafka producer and consumer, we can run the producer and consumer in separate Node.js processes to send and receive messages.

To run the producer, save the producer code in a file (e.g., producer.js) and run the file using Node.js:

node producer.js

To run the consumer, save the consumer code in a file (e.g., consumer.js) and run the file using Node.js:

node consumer.js

You should see the message "Hello, Kafka!" printed in the console by the consumer, indicating that the consumer successfully received the message sent by the producer.

In this tutorial, we have learned how to create a Kafka producer and consumer in Node.js using the kafka-node library. You can now start building real-time data streaming applications using Apache Kafka and Node.js.