In this article, I will try to give you various information from Apache Kafka installation to Kafka's own components and we will do this together with its applications.
What is the Apache Kafka?
Apache Kafka is one of the most popular technologies in the big data world.In short, it is a high-performance distributed messaging system.Developed by LinkedIn engineers and put on GitHub as Open Source in 2010, Kafka was proposed as an Apache Software Foundation Incubator Project in 2011 and was renamed Apache Kafka in 2012.
a)Apache Kafka Components
Producer
Topic
Partition
Offset
Consumer
Broker
Zookeeper
1)Producer
It is the unit that forwards the messages to the topics.
2) Topic
This is where incoming messages are stored.There can be more than one topic.You can think of it like tables in a database.
3)Partition
Kafka topics are partitioned within themselves.Partitions allow the parallelization of topics by dividing the data into specific sections.

4)Offset
Each record in partitions is assigned to certain offsets and they are placed in the order in which they come.
5)Consumer
It is the unit that reads messages from topics. There may be more than one.
6)Broker
Each Kafka server is called a Broker. Having more than one broker is important in order to make the system high availability. A leading broker is selected and in case of a crash of this broker, other brokers step in and ensure the stable operation of the system.
7)Zookeeper
Zookeeper runs as an informative service on distributed systems.In the Kafka Cluster environment, it enables Kafka components to communicate with each other.
b)Apache Kafka Setup
First, we start by installing java.
yum install java-1.8.0-openjdk.x86_64
Then we check the java version.
java -version

We add the JAVA_HOME and JRE_HOME environment variables to the /etc/bashrc file.
vi /etc/bashrc
export JRE_HOME=/usr/lib/jvm/jre
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
PATH=$PATH:$JRE_HOME:$JAVA_HOME
To activate, we cite our file as the source.
source /etc/bashrc
Now we start the kafka installation and create a user.
useradd kafka -m
We assign a password to this user.
passwd kafka
sudo usermod -aG wheel kafka
Let's log in to the user we created.
su - kafka
We go to our home directory and install kafka.Here I want to show you how to install kafka offline.
cd /Downloads
mv kafka_2.12-3.1.1.tgz /home/kafka
cd /home/kafka
tar -xvzf kafka_2.12-2.1.0.tgz
mv kafka_2.12-2.1.0/* .
rmdir /home/kafka/kafka_2.12-2.1.0
Let's create a service for Zookeeper.
sudo vi /lib/systemd/system/zookeeper.service
[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/bin/zookeeper-server-start.sh /home/kafka/config/zookeeper.properties
ExecStop=/home/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target

Now we are creating a service for Kafka.
sudo vi /etc/systemd/system/kafka.service
[Unit]
Requires=network.target remote-fs.target zookeeper.service
After=network.target remote-fs.target zookeeper.service
[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/bin/kafka-server-start.sh /home/kafka/config/server.properties
ExecStop=/home/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target

Let's edit Kafka configurations.
vi /home/kafka/config/server.properties
listeners=PLAINTEXT://:9092
log.dirs=/var/log/kafka-logs
Enter the code below to save the services we created.
systemctl daemon-reload
Let's create the Kafka log file and authorize the kafka user.
sudo mkdir -p /var/log/kafka-logs
chown kafka:kafka -R /var/log/kafka-logs
Enter the codes below to automate the services
systemctl enable zookeeper.service
systemctl enable kafka.service
Now let's start our services in order.
systemctl start zookeeper.service
systemctl start kafka.service
Let's check the status of the started services.
systemctl status zookeeper.service

systemctl status kafka.service

c)Apache Kafka Producer-Consumer App
We create a topic named data4tech by switching to Kafka user.

Let's create a producer message system on the console.
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic data4tech

Let's create a consumer message system on the console.
bin/kafka-console-consumer.sh --broker-list localhost:9092 --topic data4tech --from-beginning

Now, let's do a test about receiving the messages we sent from the producer by the consumer.

As you can see, the messages we send from the producer reach the consumer immediately.
In this article, we discussed Apache Kafka with you.If you liked the article, you can support us by sharing.
Hope to see you in new posts, take care.
...
Comments