node-red | MQTT | Kafka

lets spin off

we will explore node-red with mqtt and kafka with simple working example

node red by default will be running the mqtt broker on port 1883.
1. mqtt-in as shown in the above will subscribe to topic/test and push to kafka producer.

mqtt-in

we will be running kafka and zookeeper in container
2. kafka producer will connect to the broker on 29092 port

kafka-producer

3. we will be having consumer which will read from broker

kafka-consumer

4. finally will print the output via msg.payload

debug-console.

before all this lets run the docker-compose file for spinning up kafka
5. docker-compose.yaml

docker-compose.yaml

$ docker-compose up -d
this should spin up the docker services.

6. lets use the paho mqtt python as client to keep on publishing data to mqtt topic.

paho mqtt client

this will keep on continuously send data to topic/test mqtt.

7. Verifying data coming into the kafka consumer

node red debug logs

verifying using mqttlens chrome extension

mqtt-lens chrome extension

thats all folks, take it easy :)