Elastic Stack is a group of open source software that provides logging, searching and analytics features. Here, you will see how to install Elastic Stack on CentOS / RHEL 8.
You may refer to our previous article, if you want to install Elastic Stack on CentOS 7.
Table of Contents:
- What is Elasticsearch?
- What is Kibana?
- What is Logstash?
- What are Beats?
- What is Elastic Stack?
- Environment Specification
- Update Software Packages in Linux Server
- Install Java Development Kit (JDK) on CentOS / RHEL 8
- Install Elastic Yum Repository on CentOS / RHEL 8
- Installing Elasticsearch on CentOS / RHEL 8
- Installing Kibana on CentOS / RHEL 8
- Installing Logstash on CentOS / RHEL 8
- Installing Beats on CentOS / RHEL 8
- Installing APM Server on CentOS / RHEL 8
- Conclusion
What is Elasticsearch?:
Elasticsearch is a open source search engine based on Lucene library. Elasticsearch provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java, following an open-core business model, parts of the software are licensed under various open-source licenses (mostly the Apache License), while other parts fall under the proprietary (source-available) Elastic License. Official clients are available in Java, .NET (C#), PHP, Python, Apache Groovy, Ruby and many other languages. According to the DB-Engines ranking, Elasticsearch is the most popular enterprise search engine followed by Apache Solr, also based on Lucene.
What is Kibana?:
Kibana is an open source data visualization dashboard for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data. The source code of Kibana is available under the Apache version 2.0 software license.
Kibana also provides a presentation tool, referred to as Canvas, that allows users to create slide decks that pull live data directly from Elasticsearch.
Kibana's tight integration with Elasticsearch and the larger Elastic Stack make it ideal for supporting the following: Searching, viewing, and visualizing data indexed in Elasticsearch and analyzing the data through the creation of bar charts, pie charts, tables, histograms, and maps.
What is Logstash?:
Logstash is a tool to collect, process, and forward events and log messages. Collection is accomplished via configurable input plugins including raw socket/packet communication, file tailing, and several message bus clients. Once an input plugin has collected data it can be processed by any number of filters which modify and annotate the event data. Finally Logstash routes events to output plugins which can forward the events to a variety of external programs including Elasticsearch, local files and several message bus implementations.
Various applications send log events to Logstash, which gathers the messages, converts them into JSON documents, and stores them in an Elasticsearch cluster.
What are Beats?:
Beats is a free and open platform for single-purpose data shippers. Due to their lightweight nature, they efficiently send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch.
The Beats family consist of Filebeat, Metricbeat, Packetbeat, Winlogbeat, Auditbeat, Heartbeat and Functionbeat. Each Beats family member performs logs shipping about a respective aspect of server that you would like to monitor.
What is Elastic Stack?:
Elastic Stack (formerly the "ELK Stack") is the combination of Elasticsearch, Logstash, and Kibana, it is available as a product or service developed and maintained by Elastic N.V..
Elastic Stack uses Logstash to provide an input stream to Elasticsearch for storage and search, and Kibana accesses the data for visualizations such as dashboards. Elastic uses Beats packages to ship various kind of logs to Logstash or Elasticsearch.
Environment Specification:
We are using a minimal CentOS 8 virtual machine with following specifications.
- CPU - 3.4 Ghz (4 cores)
- Memory - 4 GB
- Storage - 40 GB
- Operating System - CentOS Linux 8.3
- Hostname – elastic-stack.centlinux.com
- IP Address - 192.168.116.230 /24
Update Software Packages in Linux Server:
Use a ssh client to connect with elastic-stack.centlinux.com server as root user.
It is a best practice to update existing software packages in your Linux operating system before installing any new software thereon.
Therefore, if your Linux server is not updated yet then you can execute the following command to update it.
# dnf update -y
After updating software packages, verify the Linux operating system and Kernel version.
# cat /etc/redhat-release CentOS Linux release 8.3.2011 # uname -r 4.18.0-240.1.1.el8_3.x86_64
Install Java Development Kit (JDK) on CentOS / RHEL 8:
Elasticsearch software is written in Java, therefore it requires Java runtime environment for execution.
You can either install Oracle JDK on CentOS 8 or use the open source alternative OpenJDK on your Linux server.
# dnf install -y java-11-openjdk
After installation check the version of Java.
# java -version
openjdk version "11.0.9" 2020-10-20 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.9+11-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.9+11-LTS, mixed mode, sharing)
Install Elastic Yum Repository on CentOS / RHEL 8:
All Elastic Stack software are provided through a common Elastic yum repository. If you add this yum repository in your Linux server then, you can install complete Elastic Stack very easily.
Import the GPG key of the Elastic yum repository using rpm command.
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Create a yum repository file in /etc/yum.repos.d directory.
# vi /etc/yum.repos.d/elasticsearch.repo
And add following directives therein.
[elasticsearch] name=Elasticsearch repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
You can use the baseurl=https://artifacts.elastic.co/packages/oss-7.x/yum if you wish to install only open source components of Elastic Stack.
Build yum cache for Elastic repository.
# dnf makecache
CentOS Linux 8 - AppStream 1.1 kB/s | 4.3 kB 00:03
CentOS Linux 8 - BaseOS 1.6 kB/s | 3.9 kB 00:02
CentOS Linux 8 - Extras 541 B/s | 1.5 kB 00:02
Elasticsearch repository for 7.x packages 262 kB/s | 19 MB 01:12
Metadata cache created.
Elastic yum repository has been installed successfully.
Installing Elasticsearch on CentOS / RHEL 8:
Since, you have setup the Elastic yum repository. Therefore, you can install the latest stable release of Elasticsearch by using the dnf command.
# dnf install -y elasticsearch
At the time of this writing, the Elasticsearch 7.10.1 is available. You must ensure that you have installed the same versions of the other Elastic Stack members for better compatibility.
If you are installing on a non-production server with limited memory, then you should reduced the Java memory pool size to run Elasticsearch in a limited memory server. Edit the jvm.options file in vim text editor.
# vi /etc/elasticsearch/jvm.options
Find the following settings in this file.
-Xms1g -Xmx1g
And update with the following values.
-Xms256m -Xmx512m
Enable and start Elasticsearch service.
# systemctl enable --now elasticsearch.service
To verify that the Elasticsearch is configured successfully, you can execute the following command.
# curl -X GET "localhost:9200/?pretty"
{
"name" : "elastic-stack.centlinux.com",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "KdBYBVSVT8aZ7DqJCrQayQ",
"version" : {
"number" : "7.10.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "1c34507e66d7db1211f66f3513706fdf548736aa",
"build_date" : "2020-12-05T01:00:33.671820Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Elasticsearch has been installed and configured successfully.
Installing Kibana on CentOS / RHEL 8:
Just like Elasticsearch, you can also install Kibana software from the same Elastic yum repository. You can use the dnf command as follows.
# dnf install -y kibana
After successful installation of Kibana software, you are required to configure it for use.
Kibana configuration file is located at /etc/kibana/kibana.yml. You can either find and update the required settings or execute the following script to configure Kibana settings in one go.
# cat >> /etc/kibana/kibana.yml << EOF > server.port: 5601 > server.host: "0.0.0.0" > server.name: "elastic-stack.centlinux.com" > elasticsearch.hosts: ["http://localhost:9200"] > EOF
Create a Linux user to own Kibana software files and processes.
# useradd kibana
Change ownership of the following directory.
# chown -R kibana:kibana /usr/share/kibana/* # chown -R kibana:kibana /var/lib/kibana/
Enable and start Kibana service.
# systemctl enable --now kibana.service
Kibana service listens on default port 5601/tcp.
To make Kibana service usable for the network computers, you have to allow incoming traffic to this port in Linux firewall.
Execute the following commands to allow Kibana service port in Linux firewall.
# firewall-cmd --permanent --add-port=5601/tcp success # firewall-cmd --reload success
Open URL http://elastic-stack.centlinux.com:5601 in a web browser.
If you see the above web page then your Kibana software has been installed and configured successfully.
Installing Logstash on CentOS / RHEL 8:
Logstash is also available in Elastic yum repository and you can execute dnf command to install it on your Linux server.
# dnf install -y logstash
Logstash can be run with default configurations, you are only required to enable and start the service by using systemctl command.
# systemctl enable --now logstash.service
Installing Beats on CentOS / RHEL 8:
For the sake of demonstration, we are only installing Filebeat on our Elastic Stack server. However, you can install any other member of Beats family by using same procedure.
Beats are also available in Elastic yum repository. Therefore use dnf command and install it on your Linux servers that you want to monitor via Elastic Stack.
# dnf install -y filebeat
Add the system module to examine the local system logs.
# filebeat modules enable system
Enabled system
Run the filebeat setup. It will scan your local system and connect itself with Kibana dashboard.
# filebeat setup
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
See more: https://www.elastic.co/guide/en/machine-learning/current/index.html
Loaded machine learning job configurations
Loaded Ingest pipelines
Enable and start Filebeat service.
# systemctl enable --now filebeat.service
Click on the Logs link under Elastic Observatory menu.
Installing APM Server on CentOS / RHEL 8:
APM (Application Performance Monitoring) Server is the new entrant in Elastic Stack.
APM Server is an optional component, but it is recommended that you should install it alongwith Elastic Stack to monitor performance of your application servers and identify the bottlenecks therein.
Since, we already have all the system logs collected in our Elasticsearch database, therefore, installing APM server adds a analytical frontend in Elastic Observatory to pinpoint the actual cause of performance bottlenecks.
APM server is also available in Elastic yum repository. Therefore, install it by using dnf command.
# dnf install -y apm-server
Enable and start APM Service.
# systemctl enable --now apm-server.service
Conclusion:
We have successfully installed Elastic Stack (Elasticsearch, Kibana, Logstash, Beats and APM Server) on CentOS / RHEL 8.
thanks for the guide - for oss options be sure to append -oss to the dnf install commands
ReplyDelete