Install Apache Kafka Server on CentOS 8

Apache Kafka is an open source stream processing software. In this article, you will learn how to install Apache Kafka Server on CentOS 8.

 

Table of Contents:

 

What is Apache Kafka? :

Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

Kafka can connect to external systems (for data import/export) via Kafka Connect and provides Kafka Streams, a Java stream processing library. Kafka uses a binary TCP-based protocol that is optimized for efficiency and relies on a "message set" abstraction that naturally groups messages together to reduce the overhead of the network roundtrip. This "leads to larger network packets, larger sequential disk operations, contiguous memory blocks which allows Kafka to turn a bursty stream of random message writes into linear writes." (Source: Wikipedia)

 

Environment Specification:

We are using a minimal CentOS 8 KVM machine with following specifications.

• CPU - 3.4 Ghz (2 cores)
• Memory - 2 GB
• Storage - 20 GB
• Operating System - CentOS 8.2
• Hostname – kafka-01.centlinux.com
• IP Address - 192.168.116.234 /24

 

Updating Software Packages in Linux Server:

Connect with kafka-01.centlinux.com as root user with the help of a ssh client.

Update installed sofware packages on your Linux server. We are using CentOS Linux in this installation guide, therefore, you can use dnf command for this purpose.

[root@kafka-01 ~]# dnf update -y
Last metadata expiration check: 0:19:21 ago on Mon 09 Nov 2020 09:21:21 PM PKT.
Dependencies resolved.
Nothing to do.
Complete!

Check the Linux operating system and Kernel version that was used in this installation guide.

[root@kafka-01 ~]# uname -r
4.18.0-193.28.1.el8_2.x86_64
[root@kafka-01 ~]# cat /etc/redhat-release
CentOS Linux release 8.2.2004 (Core)

 

Installing Java Development Kit (JDK) on Linux Server:

Apache Kafka is built using Java programming language, therefore it requires Java Development Kit 8 or later.

JDK 11 is available in standard yum repositories, therefore, you can install JDK 11 by using following Linux command.

[root@kafka-01 ~]# dnf install -y java-11-openjdk
...
Installed:
  abattis-cantarell-fonts-0.0.25-4.el8.noarch
  adwaita-cursor-theme-3.28.0-2.el8.noarch
  adwaita-icon-theme-3.28.0-2.el8.noarch
  alsa-lib-1.2.1.2-3.el8.x86_64
  at-spi2-atk-2.26.2-1.el8.x86_64
  at-spi2-core-2.28.0-1.el8.x86_64
  atk-2.28.1-1.el8.x86_64
  avahi-libs-0.7-19.el8.x86_64
  cairo-1.15.12-3.el8.x86_64
  cairo-gobject-1.15.12-3.el8.x86_64
  colord-libs-1.4.2-1.el8.x86_64
  copy-jdk-configs-3.7-1.el8.noarch
  cups-libs-1:2.2.6-33.el8.x86_64
  dconf-0.28.0-3.el8.x86_64
  fontconfig-2.13.1-3.el8.x86_64
  fontpackages-filesystem-1.44-22.el8.noarch
  fribidi-1.0.4-8.el8.x86_64
  gdk-pixbuf2-2.36.12-5.el8.x86_64
  gdk-pixbuf2-modules-2.36.12-5.el8.x86_64
  giflib-5.1.4-3.el8.x86_64
  glib-networking-2.56.1-1.1.el8.x86_64
  graphite2-1.3.10-10.el8.x86_64
  gsettings-desktop-schemas-3.32.0-4.el8.x86_64
  gtk-update-icon-cache-3.22.30-5.el8.x86_64
  gtk3-3.22.30-5.el8.x86_64
  harfbuzz-1.7.5-3.el8.x86_64
  hicolor-icon-theme-0.17-2.el8.noarch
  jasper-libs-2.0.14-4.el8.x86_64
  java-11-openjdk-1:11.0.9.11-0.el8_2.x86_64
  java-11-openjdk-headless-1:11.0.9.11-0.el8_2.x86_64
  javapackages-filesystem-5.3.0-1.module_el8.0.0+11+5b8c10bd.noarch
  jbigkit-libs-2.1-14.el8.x86_64
  json-glib-1.4.4-1.el8.x86_64
  lcms2-2.9-2.el8.x86_64
  libX11-1.6.8-3.el8.x86_64
  libX11-common-1.6.8-3.el8.noarch
  libXau-1.0.8-13.el8.x86_64
  libXcomposite-0.4.4-14.el8.x86_64
  libXcursor-1.1.15-3.el8.x86_64
  libXdamage-1.1.4-14.el8.x86_64
  libXext-1.3.3-9.el8.x86_64
  libXfixes-5.0.3-7.el8.x86_64
  libXft-2.3.2-10.el8.x86_64
  libXi-1.7.9-7.el8.x86_64
  libXinerama-1.1.4-1.el8.x86_64
  libXrandr-1.5.1-7.el8.x86_64
  libXrender-0.9.10-7.el8.x86_64
  libXtst-1.2.3-7.el8.x86_64
  libdatrie-0.2.9-7.el8.x86_64
  libepoxy-1.5.3-1.el8.x86_64
  libfontenc-1.1.3-8.el8.x86_64
  libgusb-0.3.0-1.el8.x86_64
  libjpeg-turbo-1.5.3-10.el8.x86_64
  libmodman-2.0.1-17.el8.x86_64
  libproxy-0.4.15-5.2.el8.x86_64
  libsoup-2.62.3-1.el8.x86_64
  libthai-0.1.27-2.el8.x86_64
  libtiff-4.0.9-17.el8.x86_64
  libwayland-client-1.17.0-1.el8.x86_64
  libwayland-cursor-1.17.0-1.el8.x86_64
  libwayland-egl-1.17.0-1.el8.x86_64
  libxcb-1.13.1-1.el8.x86_64
  lksctp-tools-1.0.18-3.el8.x86_64
  lua-5.3.4-11.el8.x86_64
  pango-1.42.4-6.el8.x86_64
  pixman-0.38.4-1.el8.x86_64
  rest-0.8.1-2.el8.x86_64
  ttmkfdir-3.0.9-54.el8.x86_64
  tzdata-java-2020d-1.el8.noarch
  xorg-x11-font-utils-1:7.5-40.el8.x86_64
  xorg-x11-fonts-Type1-7.5-19.el8.noarch

Complete!

 

Installing Apache Kafka Server on CentOS 8:

Apache Kafka is distributed under Apache License 2.0, therefore you can download Kafka software from offical website.

Copy the URL of your required version of Apache Kafka software from this webpage.

Use the copied URL with wget command to download the Apache Kafka software directly from Linux command line.

[root@kafka-01 ~]# cd /tmp
[root@kafka-01 tmp]# wget https://downloads.apache.org/kafka/2.6.0/kafka_2.13-2.6.0.tgz
--2020-11-09 21:52:02--  https://downloads.apache.org/kafka/2.6.0/kafka_2.13-2.6.0.tgz
Resolving downloads.apache.org (downloads.apache.org)... 88.99.95.219, 2a01:4f8:10a:201a::2
Connecting to downloads.apache.org (downloads.apache.org)|88.99.95.219|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 65537909 (63M) [application/x-gzip]
Saving to: âkafka_2.13-2.6.0.tgzâ

kafka_2.13-2.6.0.tg 100%[===================>]  62.50M   457KB/s    in 4m 55s

2020-11-09 21:56:58 (217 KB/s) - âkafka_2.13-2.6.0.tgzâ saved [65537909/65537909]

Extract downloaded tarball by using tar command.

[root@kafka-01 tmp]# tar xzf kafka_2.13-2.6.0.tgz

Now, move the extracted files to /opt/kafka directory.

[root@kafka-01 tmp]# mv kafka_2.13-2.6.0 /opt/kafka

Current versions of Apache Kafka requires Zookeeper service for distributed configurations. However, it is mentioned in Kafka documentation that "Soon, ZooKeeper will no longer be required by Apache Kafka."

But for now, you have to configure Apache Zookeeper service before configuring Kafka server.

Zookeeper binary scripts are provided with Apache Kafka setup files. You can use it to configure ZooKeeper server.

Create a systemd service unit for Apache Zookeeper.

[root@kafka-01 tmp]# cd /opt/kafka/
[root@kafka-01 kafka]# vi /etc/systemd/system/zookeeper.service

Add following directived in this file.

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/bin/bash /opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/usr/bin/bash /opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Similarly, create a systemd unit for Apache Kafka service.

[root@kafka-01 kafka]# vi /etc/systemd/system/kafka.service

Add following directives therein.

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk"
ExecStart=/usr/bin/bash /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/usr/bin/bash /opt/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Enable and start Apache Zookeeper and Kafka services.

[root@kafka-01 kafka]# systemctl daemon-reload
[root@kafka-01 kafka]# systemctl enable --now zookeeper.service
Created symlink /etc/systemd/system/multi-user.target.wants/zookeeper.service â /etc/systemd/system/zookeeper.service.
[root@kafka-01 kafka]# systemctl enable --now kafka.service
Created symlink /etc/systemd/system/multi-user.target.wants/kafka.service â /etc/systemd/system/kafka.service.

Verify status of Apache Kafka service.

[root@kafka-01 kafka]# systemctl status kafka.service
â kafka.service - Apache Kafka Server
   Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: d>
   Active: active (running) since Mon 2020-11-09 22:09:53 PKT; 58s ago
     Docs: http://kafka.apache.org/documentation.html
 Main PID: 2373 (java)
    Tasks: 65 (limit: 5879)
   Memory: 310.2M
   CGroup: /system.slice/kafka.service
           ââ2373 /usr/lib/jvm/jre-11-openjdk/bin/java -Xmx1G -Xms1G -server -X>

Nov 09 22:10:11 kafka-01.centlinux.com bash[2373]: [2020-11-09 22:10:11,661] IN>
Nov 09 22:10:12 kafka-01.centlinux.com bash[2373]: [2020-11-09 22:10:12,249] IN>
Nov 09 22:10:12 kafka-01.centlinux.com bash[2373]: [2020-11-09 22:10:12,560] IN>
Nov 09 22:10:12 kafka-01.centlinux.com bash[2373]: [2020-11-09 22:10:12,569] IN>
Nov 09 22:10:12 kafka-01.centlinux.com bash[2373]: [2020-11-09 22:10:12,644] IN>
Nov 09 22:10:12 kafka-01.centlinux.com bash[2373]: [2020-11-09 22:10:12,644] IN>
Nov 09 22:10:12 kafka-01.centlinux.com bash[2373]: [2020-11-09 22:10:12,801] IN>
Nov 09 22:10:12 kafka-01.centlinux.com bash[2373]: [2020-11-09 22:10:12,801] IN>
Nov 09 22:10:12 kafka-01.centlinux.com bash[2373]: [2020-11-09 22:10:12,801] IN>
Nov 09 22:10:12 kafka-01.centlinux.com bash[2373]: [2020-11-09 22:10:12,803] IN>

 

Create a Topic in Apache Kafka Server:

Create a topic in your Apache Kafka server.

[root@kafka-01 kafka]# /opt/kafka/bin/kafka-topics.sh --create --topic centlinux --bootstrap-server localhost:9092
Created topic centlinux.

To view the details of the topic, you can use run following script at the Linux command line.

[root@kafka-01 kafka]# /opt/kafka/bin/kafka-topics.sh --describe --topic centlinux --bootstrap-server localhost:9092
Topic: centlinux        PartitionCount: 1       ReplicationFactor: 1    Configs: segment.bytes=1073741824
        Topic: centlinux        Partition: 0    Leader: 0       Replicas: 0    Isr: 0

Add some sample events in your topic.

[root@kafka-01 kafka]# /opt/kafka/bin/kafka-console-producer.sh --topic centlinux --bootstrap-server localhost:9092
>This is the First event.
>This is the Second event.
>This is the Third event.
>^C[root@kafka-01 kafka]#

To view all the events that are inserted into a topic, you can execute following script at Linux command line.

[root@kafka-01 kafka]# /opt/kafka/bin/kafka-console-consumer.sh --topic centlinux --from-beginning --bootstrap-server localhost:9092
This is the First event.
This is the Second event.
This is the Third event.
^CProcessed a total of 3 messages

Apache Kafka server is installed on Linux server and ready for use.

 

Conclusion:

You have successfully installed Apache Kafka server on CentOS 8. To improve your skills in this area, we recommend that you should read Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale 1st Edition by O'Reilly Media.