theglobalsolutions

Apache Kafka and Zookeeper Clusters Powered by GlobalSolutions

Clusters

Apache Kafka is a publish-subscribe based fault-tolerant messaging system. In comparison to other messaging systems, Kafka has better throughput,
built-in partitioning, replication, and inherent fault-tolerance, which makes it a good fit for large-scale message processing applications.

We have packaged and hardened the Kafka service offering so our customers can seamlessly subscribe to this offering with no setup work from their end.
All that you have to do is subscribe to this offering from the marketplace, create topics, publish messages from the producers and start consuming with the consumers.

Kafka Multi Broker Clusters

Kafka broker 1 is under C:\kafka\config\server.properties

Kafka broker 2 is under C:\kafka\config\server1.properties

Kafka broker 3 is under C:\kafka\config\server2.properties

Zookeeper Multi Node Clusters

Zookeeper node 1 is under C:\Zookeeper

Zookeeper node 2 is under C:\Zookeeper_1

zookeeper node 3 is under C:\Zookeeper_2

All the three Zookeeper servers are up and running as we have installed it as services. One can start and stop the Zookeeper services (Zookeeper, Zookeeper_1, Zookeeper_2) by going to the Administrative control panel .

In the same way, all the brokers are up and running and we have installed it as services. One can start and stop the Kafka brokers( Kafka_Broker_1,Kafka_Broker_2,Kafka_Broker_3)by going to the Administrative control panel.

Run the following command to create a topic : sample

1. Create a topic with the name “sample” and a replication factor of 3, as we have three Kafka servers running.
2. Open a new command prompt by clicking on command prompt icon or by going to C:\kafka\bin\windows>
3. Type the following command and hit enter

C:\kafka\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181,localhost:2182,localhost:2183 --replication-factor 3 --partitions 1 --topic sample

Steps to publish a message to the topic.

You can create a producer from the command line by running the following command.It expects the Kafka server’s hostname and port, along with a topic name as arguments,and this will be producer command prompt.

C:\kafka\bin\windows>kafka-console-producer.bat --broker-list localhost:9092,localhost:9093,localhost:9094 --topic sample

Steps to consume the message as a consumer.

You can create a Kafka consumer from the command line by running the following command . It expects the hostname and port, along with a topic name as arguments, and this will be consumer command prompt.

C:\kafka\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --topic sample

How to check :

Now type anything in the producer command prompt and press Enter, and you should be able to see the same messages in the consumer command prompt.

Check fault-tolerant by killing one of the zookeeper services

The next step is to check the fault-tolerant.

For that, we have to kill one of the zookeeper services and check whether we were able to send and receive the messages with the help of other zookeeper servers.

Now by going to Administrative tools -> Services and stop any one of the following services, zookeeper , zookeeper_1 , or zookeeper_2 . In order to check again go to producer command prompt and send the messages, you have to receive the same messages on the consumer.

How to connect to kafka from outside? Or I am not able to publish or consume messages from Kafka?

You need to make the following changes in the server.properties ,server1.properties, and server2.properties files with the right domain name and port. Please open those files which is under C:\kafka\config\ . Locate the following property and change the value of the property to your public DNS. The plaintext in the property has to be left as is, just change the public DNS of your Ec2 instance which you can get it from AWS console.

advertised.listeners=PLAINTEXT://your.host.name:9092

Support

Please contact us at support@theglobalsolutions.net for any questions on this offering in AWS Marketplace.