Kafka Nodejs Example with Producers and Consumers

Let’s take a look at a Kafka Nodejs example with Producers and Consumers. We will be creating a kafka producer and consumer in Nodejs. If you haven’t already, check out my previous tutorial on how to setup Kafka in docker. I already created a topic called cat that I will be using.

Kafka Producer

We are going to use the npm module called kafka-node to interface with our kafka and zookeeper. The kafka is running on the local system so we don’t have to worry about pointing to any external IP addresses.

I am building a simple producer that will send a message saying I have {X} amount of cats every five seconds. {X} will increment by one each time.

var kafka = require("kafka-node"),
  Producer = kafka.Producer,
  client = new kafka.KafkaClient(),
  producer = new Producer(client);

let count = 0;

producer.on("ready", function() {
  console.log("ready");
  setInterval(function() {
    payloads = [
      { topic: "cat", messages: `I have ${count} cats`, partition: 0 }
    ];

    producer.send(payloads, function(err, data) {
      console.log(data);
      count += 1;
    });
  }, 5000);
});

producer.on("error", function(err) {
  console.log(err);
});

Pretty self-explanatory.

Kafka Consumer

Now let’s make a Kafka consumer in nodejs that will consume that message and print it.

var kafka = require("kafka-node"),
  Consumer = kafka.Consumer,
  client = new kafka.KafkaClient(),
  consumer = new Consumer(client, [{ topic: "cat", partition: 0 }], {
    autoCommit: false
  });

consumer.on("message", function(message) {
  console.log(message);
  
/**    { topic: 'cat',
  value: 'I have 385 cats',
  offset: 412,
  partition: 0,
  highWaterOffset: 413,
  key: null }

  */

});

That will consume and print the message. See, setting up a kafka producer and consumer in nodejs is pretty easy.