Apache Kafka Clusters Powered by GlobalSolutions

Apache Kafka is a publish-subscribe based fault-tolerant messaging system. In comparison to other messaging systems, Kafka has better throughput, built-in partitioning, replication, and inherent fault-tolerance, which makes it a great fit for large-scale message processing applications.

We have packaged and hardened the Kafka Cluster service offering so our customers can seamlessly subscribe with no setup work from their end. All you have to do is subscribe to this offering from the marketplace, create topics, publish messages from producers, and start consuming with consumers.

Accessing Your AMI from AWS Marketplace

To get started with your Apache Kafka Cluster stack:

1. Subscribe: Purchase the Apache Kafka Clusters AMI from the AWS Marketplace.
2. Connect via RDP:
   • In the AWS Console, select your launched instance and click Connect.
   • Choose RDP Client, upload your .pem key file, and decrypt the password using your private key.
   • Download the .rdp file and connect using the username Administrator and the decrypted password.

Kafka Multi-Broker Cluster

This offering comes with a 3-broker Kafka cluster pre-configured and running as Windows services. The broker configuration files are located at:

Broker	Configuration File
Kafka Broker 1	C:\kafka\config\server.properties
Kafka Broker 2	C:\kafka\config\server1.properties
Kafka Broker 3	C:\kafka\config\server2.properties

All brokers are up and running as services. You can start and stop individual brokers (KafkaBroker1, KafkaBroker2, KafkaBroker3) by going to Administrative Tools → Services.

Getting Started

Sample Setup

We have pre-created a sample topic kraft-test-topic using the following command from C:\kafka:

.\bin\windows\kafka-topics.bat --create --topic kraft-test-topic --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --partitions 3 --replication-factor 3

Steps to Publish a Message

You can start a producer by running the following command from C:\kafka:

.\bin\windows\kafka-console-producer.bat --topic kraft-test-topic --bootstrap-server localhost:9092

Alternatively, click the run_producer.bat batch file on the desktop — it will automatically open a producer command prompt with the above command.

Steps to Consume a Message

You can start a consumer by running the following command from C:\kafka:

.\bin\windows\kafka-console-consumer.bat --topic kraft-test-topic --bootstrap-server localhost:9092 --from-beginning

Alternatively, click the run_consumer.bat batch file on the desktop — it will automatically open a consumer command prompt with the above command.

How to Test

With both the producer and consumer command prompts open, type anything in the producer window and press Enter. You should immediately see the same message appear in the consumer command prompt.

Testing Fault Tolerance

To verify fault tolerance, stop one of the broker services and confirm that messages can still be sent and received using the remaining brokers:

1. Go to Administrative Tools → Services and stop any one of KafkaBroker1, KafkaBroker2, or KafkaBroker3.
2. Go to the producer command prompt and send a message.
3. Confirm the same message is received in the consumer command prompt — demonstrating the cluster's fault-tolerant replication.

Connecting to Kafka from Outside

To connect to Kafka from an external client, update the advertised.listeners property in all three configuration files located under C:\kafka\config\: server.properties, server1.properties, and server2.properties. Replace your.host.name with your EC2 instance's public DNS (available in the AWS Console). Leave the PLAINTEXT protocol as is:

advertised.listeners=PLAINTEXT://your-ec2-public-dns.amazonaws.com:9092

AWS Cost Optimizer — CloudInsider

Our other popular offering is the AWS Cost Optimizer aka CloudInsider, available in AWS Marketplace. This service has helped our customers save significantly on AWS and other cloud spending. It is easy to subscribe and you can see the savings in minutes.

▶ Watch Demo Video Subscribe on AWS Marketplace

Support

For any questions or assistance with our AWS Marketplace offering, reach out to us at support@theglobalsolutions.net.