Kafka is a key component of the Kubernetes cluster since it prepares a huge volume of data in real-time. Then there is Apache Kafka, which further improves the metrics of the data in the Kubernetes cluster. Apache Kafka is an event streaming platform created by LinkedIn that was initially developed as an open-source project. Linkedin donated the event streaming platform to the Apache software foundation and it changed the name of the Kafka to Apache Kafka. There are two native computer languages used on the platform, Java and Scala. Its most important mission is to provide high latency for handling real-time data.
Kafka is made of different APIs including Producer, the Connector and the Streams, the Consumer, etc. United, these APIs increase the latency level of platforms that maintain a high volume of real-time data. It runs as cluster nodes that we call Kafka brokers and is trusted by many well-known companies such as Uber, Airbnb, etc. Initially, Kafka was just a messaging queue for distributed systems that used to work as a pub-sub model as well. However, Kafka’s main motto now is to stream data on the internet on the behalf of the companies and also store a huge amount of information and keep the records of Kubernetes applications.
Generally, Kafka stores the messages in a sequence and divides them by topics. Kafka usually brokers data between systems or enables applications to respond to stream data in real-time. We need to deploy a fully-fledged Kafka cluster in Kubernetes. This approach helps us address the need for a message broker at the core of a large number of microservices. To ensure that the streaming data system does not fail, the number of Kafka instances across the nodes needs to be increased.
So, in this guide, we are going to show you exactly how to set up and run Kafka on Kubernetes so that there is no problem with streaming data and keeping a persistent volume of the data safely in the cloud storage.
How Does Kafka Work?
Apache Kafka is a vital virtual component of Kubernetes for the Kubernetes applications that are running on a cluster. The messaging system collects data and processes them in real-time no matter how voluminous it is. Kafka is, however, a publish-subscribe platform, which works as follows:
- Producers create messages and divide them into topics and publish them.
- Then Kafka categorizes those messages or determines their topics and stores them to make them immutable.
- Then consumers search for the specific topics to become a subscriber and enjoy the messages producers are publishing.
Both producers and consumers serve applications that inform the consumers with messages regarding updates of the applications. These messages are stored and sorted by Kafka brokers based on user-defined topics.
Kafka configurations work with component management tools such as Zookeeper and Platform9 Free Tier. Kafka cannot work properly without Zookeeper because Zookeeper manages all Kafka components, including producers, brokers, cluster memberships, and consumers. Hence, we will first learn about Zookeeper and then delve deeply into Platform9 Free Tier information.
How to Deploy Zookeeper?
As discussed above, Kafka won’t work without Zookeeper so the first thing you should try is to deploy Zookeeper on your Kubernetes. Zookeeper should be deployed so that the Kafka service doesn’t keep restarting, but it can only be deployed by generating zookeeper.yml. This YAML file will schedule Zookeeper pods on the Kubernetes cluster for you, so you don’t have to do anything manually. Start the deployment by following the commands below. Copy and paste the following codes into zookeeper.yml using kublect or your preferred text editor.
| apiVersion: v1
kind: Service metadata: name: zk-s labels: app: zk-1 spec: ports: – name: client port: 2181 protocol: TCP – name: follower port: 2888 protocol: TCP – name: leader port: 3888 protocol: TCP selector: app: zk-1 — kind: Deployment apiVersion: extensions/v1beta1 metadata: name: zk-deployment-1 spec: template: metadata: labels: app: zk-1 spec: containers: – name: zk1 image: bitnami/zookeeper ports: – containerPort: 2181 env: – name: ZOOKEEPER_ID value: “1” – name: ZOOKEEPER_SERVER_1 value: zk1 |
After this, you will need to create a definition file and run kubectl create -f zookeeper.yml on your Kubernetes cluster. The next step is to create the Kafka service itself.
How to Create Kafka Service?
Here, we will create the Kafka service definition file that will manage the Kafka Broker deployments by balancing the data volume of Kafka pods. And you will find the following components on a primary kafka-service.yml file.
| apiVersion: v1
kind: Service metadata: labels: app: kafkaApp name: kafka spec: ports: – port: 9092 targetPort: 9092 protocol: TCP – port: 2181 targetPort: 2181 selector: app: kafkaApp type: LoadBalancer Save the file and create a service with the following code: kubectl create -f kafka-service.ymlvh |
Let’s move on to the next step to continue with setting up Kafka on Kubernetes.
Time to Define the Kafka Replication Controller
Generate another extra .yml file that will work as the replication controller for Kafka and the kafka-repcon.yml file will have the following elements:
—
| apiVersion: v1
kind: ReplicationController metadata: labels: app: kafkaApp name: kafka-repcon spec: replicas: 1 selector: app: kafkaApp template: metadata: labels: app: kafkaApp spec: containers: – command: – zookeeper-server-start.sh – /config/zookeeper.properties image: “wurstmeister/kafka” name: zk1 ports: – containerPort: 2181 |
Now, save the file and create it with kubectl create -f kafka-repcon.yml before going on to start the Kafka server.
How to Start the Kafka Server?
You will find the configuration settings of the Kafka server in the config/server.properties file. And since configuring the Zookeeper server is done at the beginning of the article, you can start the Kafka server right away with the following command:
kafka-server-start.sh config/server.properties
Once you have started the server, it is time to create a Kafka Topic. Like Kubernetes, Kafka has a command-line utility tool as well and it is known as kafka-topics.sh. You can create new topics on the server with this utility tool.
Next open a new window and copy and paste the command written below:
kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic Topic-Name
The topic’s name is Topic-Name, which has one partition plus a replica instance. We will now proceed to setting up a Kafka producer.
How to Start a Kafka Producer?
You will find the broker port ID in the config/server.properties file. In this context, the broker that we have used is listening to port 9092 and you can use the following command to specify the listening port directly:
kafka-console-producer.sh –topic kafka-on-kubernetes –broker-list localhost:9092 –topic Topic-Name
You can use the terminal window to attach a few messages and that will be pretty much how the reports will be generated on Kafka. Now it is time to create a Kafka consumer.
How to Begin a Kafka Consumer?
You can find the default consumer configurations in the config/consumer.properties file along with the producer properties.
To get the messages for Kafka consumer, open a new terminal window and paste the following command:
kafka-console-consumer.sh –topic Topic-Name –from-beginning –zookeeper localhost:2181
Here the –from-beginning command will specify the messages in order. Even when you access information or texts from the producer’s terminal, they will appear in the consumer’s terminal.
Go ahead and learn how to scale the Kafka cluster next.
Scaling the Kafka Cluster
Scaling your Kafka cluster is easy with the kubectl scale rc kafka-rc –replicas=6 command on the kubectl to make the cluster an administrator of Kubernetes. The process involves extending the number of pods from 1 to 6.
Somethings to Consider While Running Kafka on Kubernetes
By running Kafka on Kubernetes, you can simplify the operations such as scaling, restarts, upgrades, and monitoring the Kubernetes applications. Although there are some points you should consider while running Kafka on Kubernetes.
Low Latency Network and Storage
Kafka requires low latency and high storage that also refers to having low contention for information and high throughput. It helps Kafka to deliver fast media to brokers so that they can access the data locally and in the system where the pod is running. Therefore, it improves the overall system performance.
Availability of Kafka Brokers Should be High
The Kafka brokers can be deployed throughout the Kubernetes cluster. Brokers then connect users over fault domains. As Kubernetes can automatically correct pods when nodes and containers crash, it can also correct brokers when they go down. The high availability of brokers in Kafka Kubernetes cannot be ignored. However, one thing to consider about these brokers is what happens to the information that Kafka stores? To make sure if the data is following the pod, you need to apply a data replication strategy. You will be able to use different brokers for a higher throughput, which will also help you quickly recover damaged brokers.
Data Protection and Security
Before you Set up and Run Kafka on Kubernetes, you need to know the data protection and data security systems of Kafka. Kafka provides multiple replications of topics and monitoring of data across Kubernetes clusters. So, replication is like a backup that protects the data. When something fails in the cluster such as a node, the replication will back up the data. Likewise, mirroring of data makes the data prepared in other data centers.
There is also an in-built data security system in Kafka that implements various authentication systems such as using SSL among brokers. So, these data and filesystems are protected from manipulators or hackers on the internet.
Conclusion
We have now explained how to set up and use Kafka with Kubernetes. All you need to do is follow every step starting with deploying Zookeeper. If you are looking for a tutorial on how to set up the Platform9 Free Tier Cluster specifically for running Kafka on Kubernetes, we encourage you to check out our other posts. The rest of the information can be found in this tutorial, which will help you create the Kafka service and run the Kafka server on Kubernetes without any hassle.
You should remember that Kafka is an extremely powerful tool that’s used and supported by numerous companies, including Spotify, Coursera, Netflix, and many more. To improve the value of your organization and run complex microservices on Kubernetes clusters, you will definitely need Kafka. If any of these codes and command lines appear difficult to follow, please feel free to ask us for assistance. Furthermore, each of your comments will be responded to shortly.