Install Elastic Stack 7.2 on CentOS 7

Elastic Stack (formerly ELK Stack) is a popular Log Analytics solution consists of three open source software components i.e. Elasticsearch, Logstash and Kibana. Elastic Stack is available as a Software, a Docker based Container as well as a Service by many Cloud service providers like AWS and others.

Elasticsearch is a search engine based on Lucene library. Elasticsearch is a distributed, multitenant-capable, full-text search engine with a HTTP web interface and schema-free JSON documents.

Logstash is a server-side data processing pipeline that receives data from multiple sources simultaneously, transform it and then send it to Elasticsearch.

Kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. User can create bar, line and scatter plots, or pie charts and maps on top of large volume of data.

In this article, we are installing Elastic Stack 7.2 on CentOS 7 server. This article provides the step by step recipe to setup each component of Elastic Stack server on CentOS 7, but it does not provide any tutorials about Elastic Stack usage and development. If you are eager to learn more about Elastic Stack usage then you should read Learning Elastic Stack 7.0: Distributed search, analytics, and visualization using Elasticsearch, Logstash, Beats, and Kibana, 2nd Edition by Packt Publishing.

Elastic Stack Features
Elastic Stack System Requirements
Environment Specification
Install Elastic Stack Prerequisite - Java on CentOS 7
Installing Elasticsearch Yum Repository on CentOS 7
Installing Elasticsearch 7.2 on CentOS 7
Installing Logstash on CentOS 7
Installing Kibana 7.2 on CentOS 7
Installing Filebeat 7.2 on CentOS 7
Testing Our Elastic Stack configurations

Elastic Stack Features:

Detailed list of Elastic features are provided by Elasticsearch official website. Some of these features are being copied below.

Clustering and high availability
Automatic data rebalancing
Horizontal scalability
Full stack monitoring
Index lifecycle management
Encrypted communication
Role based access control

Elastic Stack System Requirements:

Hardware requirements for Elastic stack (Elasticsearch, Logstash and Kibana) depend upon the number of log sources and the amount of log generated. Some recommended hardware specifications are mentioned in Elasticsearch documentation.

Elastic stack requires JVM (Java Virtual Machine) to run. Therefore, we have to install a supported version of JDK (Java Development Kit) to be installed on our CentOS 7 server.

Environment Specification:

Based on Elastic stack system requirements, we have configured a CentOS 7 based virtual machine with following specification.

CPU – 3.4 Ghz (Dual Core)
Memory – 2 GB
Storage – 20 GB
Operating System – CentOS 7.6
Java Version – OpenJDK 1.8
Hostname – elasticsearch-01.example.com
IP Address – 192.168.116.187 /24

Install Elastic Stack Prerequisite - Java on CentOS 7:

Connect with elasticsearch-01.example.com using ssh as root user.

OpenJDK 8 is available in standard yum repository. Therefore, we are installing OpenJDK 8 using yum command.

[root@elasticsearch-01 ~]# yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
...
Installed:
  java-1.8.0-openjdk.x86_64 1:1.8.0.212.b04-0.el7_6
  java-1.8.0-openjdk-devel.x86_64 1:1.8.0.212.b04-0.el7_6

Dependency Installed:
  atk.x86_64 0:2.28.1-1.el7
  avahi-libs.x86_64 0:0.6.31-19.el7
  cairo.x86_64 0:1.15.12-3.el7
  cups-libs.x86_64 1:1.6.3-35.el7
  fribidi.x86_64 0:1.0.2-1.el7
  gdk-pixbuf2.x86_64 0:2.36.12-3.el7
  graphite2.x86_64 0:1.3.10-1.el7_3
  gtk-update-icon-cache.x86_64 0:3.22.30-3.el7
  gtk2.x86_64 0:2.24.31-1.el7
  harfbuzz.x86_64 0:1.7.5-2.el7
  hicolor-icon-theme.noarch 0:0.12-7.el7
  jasper-libs.x86_64 0:1.900.1-33.el7
  java-1.8.0-openjdk-headless.x86_64 1:1.8.0.212.b04-0.el7_6
  jbigkit-libs.x86_64 0:2.0-11.el7
  libXcomposite.x86_64 0:0.4.4-4.1.el7
  libXcursor.x86_64 0:1.1.15-1.el7
  libXdamage.x86_64 0:1.1.4-4.1.el7
  libXfixes.x86_64 0:5.0.3-1.el7
  libXft.x86_64 0:2.3.2-2.el7
  libXinerama.x86_64 0:1.1.3-2.1.el7
  libXrandr.x86_64 0:1.5.1-2.el7
  libXxf86vm.x86_64 0:1.1.4-1.el7
  libglvnd.x86_64 1:1.0.1-0.8.git5baa1e5.el7
  libglvnd-egl.x86_64 1:1.0.1-0.8.git5baa1e5.el7
  libglvnd-glx.x86_64 1:1.0.1-0.8.git5baa1e5.el7
  libthai.x86_64 0:0.1.14-9.el7
  libtiff.x86_64 0:4.0.3-27.el7_3
  libwayland-client.x86_64 0:1.15.0-1.el7
  libwayland-server.x86_64 0:1.15.0-1.el7
  libxshmfence.x86_64 0:1.2-1.el7
  mesa-libEGL.x86_64 0:18.0.5-4.el7_6
  mesa-libGL.x86_64 0:18.0.5-4.el7_6
  mesa-libgbm.x86_64 0:18.0.5-4.el7_6
  mesa-libglapi.x86_64 0:18.0.5-4.el7_6
  pango.x86_64 0:1.42.4-2.el7_6
  pcsc-lite-libs.x86_64 0:1.8.8-8.el7
  pixman.x86_64 0:0.34.0-1.el7

Complete!

Installing Elasticsearch Yum Repository on CentOS 7:

The procedure to install Elasticsearch Yum Repository is available in Elasticsearch documentation. You can also install yum repoositories for previous versions of Elastic stack using the same procedure.

Download and install the public signing key as follows.

[root@elasticsearch-01 ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Create a new yum configuration file to install Elasticsearch Yum Repository on CentOS 7.

[root@elasticsearch-01 ~]# cat > /etc/yum.repos.d/elasticsearch.repo << EOF
> [elasticsearch-7.x]
> name=Elasticsearch repository for 7.x packages
> baseurl=https://artifacts.elastic.co/packages/7.x/yum
> gpgcheck=1
> gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
> enabled=1
> autorefresh=1
> type=rpm-md
> EOF

Build cache for Elasticsearch Yum Repository.

[root@elasticsearch-01 ~]# yum makecache fast
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.dhakacom.com
 * extras: mirror.dhakacom.com
 * updates: mirror.dhakacom.com
base                                                     | 3.6 kB     00:00
elasticsearch-7.x                                        | 1.3 kB     00:00
extras                                                   | 3.4 kB     00:00
updates                                                  | 3.4 kB     00:00
elasticsearch-7.x/primary                                  |  31 kB   00:01
elasticsearch-7.x                                                         85/85
Metadata Cache Created

We have successfully installed Elasticsearch Yum Repository. We can now install Elastic stack components on our CentOS 7 server.

Installing Elasticsearch 7.2 on CentOS 7:

Install Elasticsearch 7.2 using yum command.

[root@elasticsearch-01 ~]# yum install -y elasticsearch
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.dhakacom.com
 * extras: mirror.dhakacom.com
 * updates: mirror.dhakacom.com
Resolving Dependencies
--> Running transaction check
---> Package elasticsearch.x86_64 0:7.2.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package             Arch         Version         Repository               Size
================================================================================
Installing:
 elasticsearch       x86_64       7.2.0-1         elasticsearch-7.x       321 M

Transaction Summary
================================================================================
Install  1 Package

Total download size: 321 M
Installed size: 511 M
Downloading packages:
elasticsearch-7.2.0-x86_64.rpm                             | 321 MB   15:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Creating elasticsearch group... OK
Creating elasticsearch user... OK
  Installing : elasticsearch-7.2.0-1.x86_64                                 1/1
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch
  Verifying  : elasticsearch-7.2.0-1.x86_64                                 1/1

Installed:
  elasticsearch.x86_64 0:7.2.0-1

Complete!

Configure JVM (Java Virtual Machine) options for Elasticsearch as follows.

[root@elasticsearch-01 ~]# vi /etc/elasticsearch/jvm.options

Find and set following parameters.

-Xms256m
-Xmx512m

Enable and start Elasticsearch service.

[root@elasticsearch-01 ~]# systemctl daemon-reload
[root@elasticsearch-01 ~]# systemctl enable elasticsearch.service
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[root@elasticsearch-01 ~]# systemctl start elasticsearch.service

Add Elasticsearch service port 9200/tcp in SELinux Policy as follows.

[root@elasticsearch-01 ~]# semanage port -m -t http_port_t 9200 -p tcp

Test Elasticsearch configuration.

[root@elasticsearch-01 ~]# curl http://127.0.0.1:9200
{
  "name" : "elasticsearch-01.example.com",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AkTQvcFiSwawa7mGqcH5hA",
  "version" : {
    "number" : "7.2.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "508c38a",
    "build_date" : "2019-06-20T15:54:18.811730Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Elasticsearch 7.2 has been installed on our CentOS 7 server.

If there is any error during startup of Elasticsearch service then check /var/log/elasticsearch/gc.log for detailed information and troubleshooting.

Installing Logstash on CentOS 7:

Logstash is used to setup a centralized log server for other servers in a network.

Logstash 7.2 is also available in Elasticsearch yum repository. Therefore, we can easily install it using yum command.

[root@elasticsearch-01 ~]# yum install -y logstash
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.dhakacom.com
 * extras: mirror.dhakacom.com
 * updates: mirror.dhakacom.com
Resolving Dependencies
--> Running transaction check
---> Package logstash.noarch 1:7.2.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package         Arch          Version           Repository                Size
================================================================================
Installing:
 logstash        noarch        1:7.2.0-1         elasticsearch-7.x        164 M

Transaction Summary
================================================================================
Install  1 Package

Total download size: 164 M
Installed size: 286 M
Downloading packages:
logstash-7.2.0.rpm                                         | 164 MB   12:35
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 1:logstash-7.2.0-1.noarch                                    1/1
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash
  Verifying  : 1:logstash-7.2.0-1.noarch                                    1/1

Installed:
  logstash.noarch 1:7.2.0-1

Complete!

Configure Logstash as follows.

[root@elasticsearch-01 ~]# cat > /etc/logstash/conf.d/logstash.conf << EOF
> input {
>  beats {
>    port => 5044
>    ssl => false
>    }
> }
>
> filter {
> if [type] == "syslog" {
>     grok {
>       match => { "message" => "%{SYSLOGLINE}" }
>     }
>
>     date {
> match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
> }
>   }
> }
>
> output {
>  elasticsearch {
>   hosts => localhost
>     index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
>        }
> stdout {
>     codec => rubydebug
>        }
> }
> EOF

Enable and start Logstash service.

[root@elasticsearch-01 ~]# systemctl enable logstash.service
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.
[root@elasticsearch-01 ~]# systemctl start logstash.service

Check /var/log/logstash/logstash-plain.log for troubleshooting Logstash service errors.

Allow Logstash service port in Linux Firewall.

[root@elasticsearch-01 ~]# firewall-cmd --permanent --add-port=5044/tcp
success
[root@elasticsearch-01 ~]# firewall-cmd --reload
success

Logstash 7.2 has been installed and configured on our CentOS 7 server.

Installing Kibana 7.2 on CentOS 7:

Kibana 7.2 can be installed from Elasticsearch yum repository using yum command.

[root@elasticsearch-01 ~]# yum -y install kibana
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.dhakacom.com
 * extras: mirror.dhakacom.com
 * updates: mirror.dhakacom.com
Resolving Dependencies
--> Running transaction check
---> Package kibana.x86_64 0:7.2.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package        Arch           Version          Repository                 Size
================================================================================
Installing:
 kibana         x86_64         7.2.0-1          elasticsearch-7.x         209 M

Transaction Summary
================================================================================
Install  1 Package

Total download size: 209 M
Installed size: 532 M
Downloading packages:
kibana-7.2.0-x86_64.rpm                                    | 209 MB   09:40
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : kibana-7.2.0-1.x86_64                                        1/1
  Verifying  : kibana-7.2.0-1.x86_64                                        1/1

Installed:
  kibana.x86_64 0:7.2.0-1

Complete!

Configure Kibana settings as follows.

[root@elasticsearch-01 ~]# cat >> /etc/kibana/kibana.yml << EOF
> server.port: 5601
> server.host: "0.0.0.0"
> server.name: "elasticsearch-01.example.com"
> elasticsearch.hosts: ["http://localhost:9200"]
> EOF

Enable and start Kibana service.

[root@elasticsearch-01 ~]# systemctl enable --now kibana
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.

Allow Kibana service port in Linux firewall.

[root@elasticsearch-01 ~]# firewall-cmd --permanent --add-port=5601/tcp
success
[root@elasticsearch-01 ~]# firewall-cmd --reload
success

Kibana 7.2 has been installed and configured on our CentOS 7 server.

Installing Filebeat 7.2 on CentOS 7:

Filebeat is an agent that sends logs to Logstash. Filebeat is also available in Elasticsearch yum repository.

Since, we are installing on the same server (elasticsearch-01.example.com), therefore, we have already installed Elasticsearch yum repository on this server. Otherwise, we have to install Elasticsearch yum repository before installing Filebeat on other CentOS 7 machines.

Install Filebeat 7.2 using yum command.

[root@elasticsearch-01 ~]# yum install -y filebeat
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.dhakacom.com
 * extras: mirror.dhakacom.com
 * updates: mirrors.psu.ac.th
base                                                     | 3.6 kB     00:00
elasticsearch-7.x                                        | 1.3 kB     00:00
extras                                                   | 3.4 kB     00:00
updates                                                  | 3.4 kB     00:00
updates/7/x86_64/primary_db                                | 6.5 MB   00:20
Resolving Dependencies
--> Running transaction check
---> Package filebeat.x86_64 0:7.2.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package         Arch          Version           Repository                Size
================================================================================
Installing:
 filebeat        x86_64        7.2.0-1           elasticsearch-7.x         21 M

Transaction Summary
================================================================================
Install  1 Package

Total download size: 21 M
Installed size: 77 M
Downloading packages:
filebeat-7.2.0-x86_64.rpm                                  |  21 MB   00:57
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : filebeat-7.2.0-1.x86_64                                      1/1
  Verifying  : filebeat-7.2.0-1.x86_64                                      1/1

Installed:
  filebeat.x86_64 0:7.2.0-1

Complete!

Edit Filebeat configuration file.

[root@elasticsearch-01 ~]# vi /etc/filebeat/filebeat.yml

Locate and enabled filebeat.input section.

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

Locate and comment all lines in output.elasticsearch section.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

Locate and uncomment output.logstash section as follows.

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

Enable and start Filebeat service.

[root@elasticsearch-01 ~]# systemctl enable --now filebeat.service
Created symlink from /etc/systemd/system/multi-user.target.wants/filebeat.service to /usr/lib/systemd/system/filebeat.service.

Filebeat 7.2 is installed and configured on the same CentOS 7 server.