Install CockroachDB Secure Cluster on CentOS 8 - CentLinux


Thursday, 5 November 2020

Install CockroachDB Secure Cluster on CentOS 8

Install CockroachDB Secure Cluster on CentOS 8

CockroachDB is a highly scalable and indestructible, distributed database management system. Here, you will learn how to install CockroachDB secure cluster on CentOS 8.


Table of Contents:


What is CockroachDB? :

CockroachDB is an elastic, indestructible SQL database for modern applications. It is developed by Cockroach Labs as an alternative to Google Spanner and available in CockroachCloud, CockroachDB Enterprise and Core editions.

CockroachDB Core edition is free and open source. Whereas, CockroachDB Enterprise edition has a licensing cost.

You may find the CockroachDB Enterprise Pricing on their official website.

CockroachDB is designed to store copies of data in multiple stores in order to deliver speedy access. Replication is automated and node addition and removal is very easy.

You can easily scale a CockroachDB cluster from a single node on your laptop to thousands of server nodes.

CockroachDB is designed to run in the cloud and be resilient to failures. The result is a database that is described as "almost impossible" to take down. Even if multiple servers or an entire datacenter were to go offline, CockroachDB would keep services online.


Environment Specification:

We are using two minimal CentOS 8 virtual machines with following specifications.

CockroachDB node 1

  • CPU - 3.4 Ghz (2 cores)
  • Memory - 2 GB
  • Storage - 20 GB
  • Operating System - CentOS 8.2
  • Hostname –
  • IP Address - /24

CockroachDB node 2

  • CPU - 3.4 Ghz (2 cores)
  • Memory - 2 GB
  • Storage - 20 GB
  • Operating System - CentOS 8.2
  • Hostname –
  • IP Address - /24


Updating Linux Server Packages:

Use a SSH client to connect with as root user.

It is a best practice to update your Linux server packages frequently, especially before installation or configuration of a new software.

With the help of dnf command, update installed packages in your Linux operating system.

[root@cockroachdb-01 ~]# dnf update -y Last metadata expiration check: 0:02:11 ago on Fri 30 Oct 2020 10:02:03 PM PKT. Dependencies resolved. Nothing to do. Complete!

Our Linux server is already up-to-date.

Check the Kernel version and Linux distribution of servers, that we are using in this installation guide.

[root@cockroachdb-01 ~]# uname -r 4.18.0-193.19.1.el8_2.x86_64 [root@cockroachdb-01 ~]# cat /etc/redhat-release CentOS Linux release 8.2.2004 (Core)


Configuring Name Resolution of CockroachDB Nodes:

Name resolution is very critical while setting up a Linux cluster. Because, if a node is unable to resolve the hostname of the other nodes, then the cluster setup will raise different type of errors.

For this purpose, you can either configure an Authoritative DNS server or simply use the local DNS resolver.

Here, we are configuring the local DNS resolver for hostname resolution of CockroachDB cluster nodes.

Edit /etc/hosts file by using vim editor.

[root@cockroachdb-01 ~]# vi /etc/hosts

Add following entries in this file. cockroachdb-01 cockroachdb-02

These entries are quiet enough to configure name resolution of our CockroachDB cluster nodes.


Setup Time Synchronization on Linux Server:

Just like any other clustering setup, CockroachDB cluster also requires time synchronization across all the nodes.

If it unable to synchronize time on any of the CockroachDB node and cause a time drift of more than 500ms, then that node won't be started until its time is synchronized with the other CockroachDB nodes.

In Red Hat based Linux distros, Chrony is the preferred NTP client/server since RHEL 7.

In this guide, we are using CentOS 8, therefore, you can install Chrony by using dnf command.

[root@cockroachdb-01 ~]# dnf install -y chrony Last metadata expiration check: 1:44:58 ago on Sun 01 Nov 2020 09:14:24 AM PKT. Dependencies resolved. ================================================================================ Package Architecture Version Repository Size ================================================================================ Installing: chrony x86_64 3.5-1.el8 BaseOS 271 k Installing weak dependencies: timedatex x86_64 0.5-3.el8 BaseOS 32 k Transaction Summary ================================================================================ Install 2 Packages Total download size: 303 k Installed size: 731 k Downloading Packages: (1/2): timedatex-0.5-3.el8.x86_64.rpm 29 kB/s | 32 kB 00:01 (2/2): chrony-3.5-1.el8.x86_64.rpm 197 kB/s | 271 kB 00:01 -------------------------------------------------------------------------------- Total 117 kB/s | 303 kB 00:02 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : timedatex-0.5-3.el8.x86_64 1/2 Running scriptlet: timedatex-0.5-3.el8.x86_64 1/2 Running scriptlet: chrony-3.5-1.el8.x86_64 2/2 Installing : chrony-3.5-1.el8.x86_64 2/2 Running scriptlet: chrony-3.5-1.el8.x86_64 2/2 Verifying : chrony-3.5-1.el8.x86_64 1/2 Verifying : timedatex-0.5-3.el8.x86_64 2/2 Installed: chrony-3.5-1.el8.x86_64 timedatex-0.5-3.el8.x86_64 Complete!

Enable and start Chrony service by using following Linux command.

[root@cockroachdb-01 ~]# systemctl enable --now chronyd.service

Execute the following command to check the NTP sources and time synchronization status.

[root@cockroachdb-01 ~]# chronyc sources -v 210 Number of sources = 4 .-- Source mode '^' = server, '=' = peer, '#' = local clock. / .- Source state '*' = current synced, '+' = combined , '-' = not combined, | / '?' = unreachable, 'x' = time may be in error, '~' = time too variable. || .- xxxx [ yyyy ] +/- zzzz || Reachability register (octal) -. | xxxx = adjusted offset, || Log2(Polling interval) --. | | yyyy = measured offset, || \ | | zzzz = estimated error. || | | \ MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^+> 2 6 17 2 -3915us[-2554us] +/- 94ms ^+> 3 6 17 2 +25ms[ +27ms] +/- 83ms ^+> 2 6 17 2 -40ms[ -39ms] +/- 134ms ^* 2 6 17 2 +3885us[+5131us] +/- 88ms


Installing CockroachDB Prerequisite Software:

CockroachDB requires some software packages, that are usually preinstalled on a minimal installed CentOS 8 operating system.

However, you can execute the dnf command to install these packages, if they are not already installed.

[root@cockroachdb-01 ~]# dnf install -y glibc ncurses-libs tzdata Last metadata expiration check: 0:16:51 ago on Fri 30 Oct 2020 10:34:28 PM PKT. Package glibc-2.28-101.el8.x86_64 is already installed. Package ncurses-libs-6.1-7.20180224.el8.x86_64 is already installed. Package tzdata-2020a-1.el8.noarch is already installed. Dependencies resolved. Nothing to do. Complete!


Installing CockroachDB Software on CentOS 8:

You can download CockroachDB Core edition from GitHub or Cockroach Labs website free of cost. Whereas, the acquisition of CockroachCloud or CockroachDB Enterprise edition requires a license fee.

You can see complete CockroachCloud and CockroachDB Enterprise pricing on their official website.

Download CockroachDB by using wget command.

[root@cockroachdb-01 ~]# cd /tmp [root@cockroachdb-01 tmp]# wget --2020-10-30 22:52:11-- Resolving (,,, ... Connecting to (||:443... connected. HTTP request sent, awaiting response... 200 OK Length: 73681901 (70M) [binary/octet-stream] Saving to: âcockroach-v20.1.8.linux-amd64.tgzâ cockroach-v20.1.8.l 100%[===================>] 70.27M 253KB/s in 5m 50s 2020-10-30 22:58:04 (206 KB/s) - âcockroach-v20.1.8.linux-amd64.tgzâ saved [73681901/73681901]

Extract downloaded Tarball using tar command.

[root@cockroachdb-01 tmp]# tar xf cockroach-v20.1.8.linux-amd64.tgz

Create directories to install CockroachDB software and related files.

[root@cockroachdb-01 tmp]# mkdir -p /opt/cockroachdb/{bin,certs,private}

Copy extracted files in /opt/cockroachdb/bin directory.

[root@cockroachdb-01 tmp]# cp -i cockroach-v20.1.8.linux-amd64/cockroach /opt/cockroachdb/bin/

Check the file permissions of the cockroach file.

[root@cockroachdb-01 tmp]# ls -al /opt/cockroachdb/bin/ total 163100 drwxr-xr-x. 2 root root 23 Oct 31 11:12 . drwxr-xr-x. 5 root root 45 Oct 31 11:11 .. -rwxr-xr-x. 1 root root 167014096 Oct 31 11:12 cockroach

Create a Linux user to own CockroachDB software and processes. Also change the ownership of CockroachDB software files to cockroach user.

[root@cockroachdb-01 tmp]# useradd -r cockroach [root@cockroachdb-01 tmp]# chown -R cockroach.cockroach /opt/cockroachdb/

Create a Soft link for cockroach file in /usr/local/bin/ directory, to make it executable from any path.

[root@cockroachdb-01 tmp]# ln -s /opt/cockroachdb/bin/cockroach /usr/local/bin/cockroach

Verify the CockroachDB version by using following command.

[root@cockroachdb-01 tmp]# cd [root@cockroachdb-01 ~]# cockroach version Build Tag: v20.1.8 Build Time: 2020/10/21 15:46:38 Distribution: CCL Platform: linux amd64 (x86_64-unknown-linux-gnu) Go Version: go1.13.9 C Compiler: gcc 6.3.0 Build SHA-1: ffd029f51aa134f2bce4a39ef1f3ad095c3856ad Build Type: release

Above steps (from start of this article) must be executed on each CockroachDB node. Whereas, the steps onwards are specific to nodes and must be executed on the mentioned nodes only.


Setup Secure CockroachDB Cluster on CentOS 8:

CockroachDB cluster can be configured in secure and insecure modes.

Configuration of CockroachDB cluster in insecure mode is pretty simple but do not enforce encryption of inter-cluster communication.

Whereas, Secure mode uses SSL/TLS certificates to enforce encryption of inter-cluster communication and authorization.

First of all, you need to create a Certificate Authority (CA), that will be used to digitally sign any certificate that you will generate for your CockroachDB secure cluster.

You can use following cockroach command to create a certificate authority, or you can also create a certificate authority with openssl command.

[root@cockroachdb-01 ~]# cockroach cert create-ca \ > --certs-dir=/opt/cockroachdb/certs \ > --ca-key=/opt/cockroachdb/private/ca.key

Generate a SSL/TLS certificate for our first CockroachDB node with the help of following command.

[root@cockroachdb-01 ~]# cockroach cert create-node \ > \ > cockroachdb-01 \ > localhost \ > --certs-dir=/opt/cockroachdb/certs \ > --ca-key=/opt/cockroachdb/private/ca.key

Generate a SSL/TLS certificate for CockroachDB client by using cockroach command.

[root@cockroachdb-01 ~]# cockroach cert create-client \ > root \ > --certs-dir=/opt/cockroachdb/certs \ > --ca-key=/opt/cockroachdb/private/ca.key

Copy the SSL/TLS certificates on other nodes of CockroachDB cluster.

[root@cockroachdb-01 ~]# scp /opt/cockroachdb/certs/* \ > root@cockroachdb-02:/opt/cockroachdb/certs/ root@cockroachdb-02's password: ca.crt 100% 1111 27.5KB/s 00:00 client.root.crt 100% 1099 857.5KB/s 00:00 client.root.key 100% 1675 1.5MB/s 00:00 node.crt 100% 1159 1.0MB/s 00:00 node.key 100% 1679 108.7KB/s 00:00

Copy the Certificate Authority key on other nodes, so we can also create SSL/TLS on other nodes.

[root@cockroachdb-01 ~]# scp /opt/cockroachdb/private/* root@cockroachdb-02:/opt/cockroachdb/private/ root@cockroachdb-02's password: ca.key 100% 1679 81.3KB/s 00:00

Connect to as root user by using ssh command.

Remove the Node certificate/key that we have generated on the cockroachdb-01 node.

[root@cockroachdb-02 ~]# rm -f /opt/cockroachdb/certs/node.*

Generate a SSL/TLS certificate for cockroachdb-02 node as follows.

[root@cockroachdb-02 ~]# cockroach cert create-node \ > \ > cockroachdb-02 \ > localhost \ > --certs-dir=/opt/cockroachdb/certs \ > --ca-key=/opt/cockroachdb/private/ca.key


Create SystemD Service for CockroachDB:

To enable auto-start of CockroachDB server during Linux startup, we need to create a systemd service unit.

Connect with as root user by using ssh command.

Create a systemd service unit file by using vim editor.

[root@cockroachdb-01 ~]# vi /etc/systemd/system/cockroachdb.service

Add following directives in this file.

[Unit] Description=Cockroach Database cluster node [Service] Type=notify WorkingDirectory=/opt/cockroachdb ExecStart=/usr/local/bin/cockroach start --certs-dir=/opt/cockroachdb/certs --advertise-addr=cockroachdb-01 --join=cockroachdb-01,cockroachdb-02 TimeoutStopSec=60 Restart=always RestartSec=10 StandardOutput=syslog StandardError=syslog SyslogIdentifier=cockroach User=cockroach [Install]

You need to replace the --advertise-addr with the hostname of the CockroachDB node on which you are creating this systemd service.

Enable and start CockroachDB service.

[root@cockroachdb-01 ~]# systemctl enable --now cockroachdb.service Created symlink /etc/systemd/system/ â /etc/systemd/system/cockroachdb.service.

Check the CockroachDB service ports to verify that the CockroachDB service is started without any error.

[root@cockroachdb-01 ~]# ss -tulpn | grep cockroach tcp LISTEN 0 128 *:8080 *:* users:(("cockroach",pid=1692,fd=9)) tcp LISTEN 0 128 *:26257 *:* users:(("cockroach",pid=1692,fd=15))


Configure Linux Firewall for CockroachDB:

CockroachDB default service ports are 8080/tcp for Web Admin UI, 26257/tcp for SQL interface.

You are required to allow both of above service ports in Linux firewall.

[root@cockroachdb-01 ~]# firewall-cmd --permanent --add-port={8080,26257}/tcp success [root@cockroachdb-01 ~]# firewall-cmd --reload success


Initialize CockroachDB Cluster on Linux Server:

CockroachDB configuration are completed, now you can execute the following command on any CockroachDB node to initialize the cluster.

[root@cockroachdb-01 ~]# cockroach init --certs-dir=/opt/cockroachdb/certs --host=cockroachdb-01:26257 Cluster successfully initialized


Accessing CockroachDB SQL Shell:

Connect to cockroachdb-01 SQL shell by using following command.

[root@cockroachdb-01 ~]# cockroach sql --certs-dir=/opt/cockroachdb/certs --host=cockroachdb-01:26257 # # Welcome to the CockroachDB SQL shell. # All statements must be terminated by a semicolon. # To exit, type: \q. # # Server version: CockroachDB CCL v20.1.8 (x86_64-unknown-linux-gnu, built 2020/10/21 15:46:38, go1.13.9) (same version as client) # Cluster ID: 50a8b514-7e6f-4a4a-936e-8a4d68aa1007 # # Enter \? for a brief introduction. # root@cockroachdb-01:26257/defaultdb>

List down available databases in CockroachDB server.

root@cockroachdb-01:26257/defaultdb> SHOW DATABASES; database_name ----------------- defaultdb postgres system (3 rows) Time: 1.742219ms

Create a new database by using CREATE statement.

root@cockroachdb-01:26257/defaultdb> CREATE DATABASE contacts; CREATE DATABASE Time: 11.39776ms

Create a new table in contacts database.

root@cockroachdb-01:26257/defaultdb> USE contacts; SET Time: 661.825µs root@cockroachdb-01:26257/contacts> CREATE TABLE emails (id INT PRIMARY KEY, email varchar(40)); CREATE TABLE Time: 164.404539ms

Insert a few rows in emails table.

root@cockroachdb-01:26257/contacts> INSERT INTO emails VALUES (1,''); INSERT 1 Time: 2.482089ms root@cockroachdb-01:26257/contacts> INSERT INTO emails VALUES (2,''); INSERT 1 Time: 2.278789ms root@cockroachdb-01:26257/defaultdb> \q

Now connect with second CockroachDB node, and check has changes been replicated to that database.

[root@cockroachdb-01 ~]# cockroach sql --certs-dir=/opt/cockroachdb/certs --host=cockroachdb-02:26257 # # Welcome to the CockroachDB SQL shell. # All statements must be terminated by a semicolon. # To exit, type: \q. # # Server version: CockroachDB CCL v20.1.8 (x86_64-unknown-linux-gnu, built 2020/10/21 15:46:38, go1.13.9) (same version as client) # Cluster ID: 50a8b514-7e6f-4a4a-936e-8a4d68aa1007 # # Enter \? for a brief introduction. # root@cockroachdb-02:26257/defaultdb> SHOW DATABASES; database_name ----------------- contacts defaultdb postgres system (4 rows) Time: 10.066515ms root@cockroachdb-02:26257/defaultdb> USE contacts; SET Time: 2.853712ms root@cockroachdb-02:26257/contacts> SELECT * FROM emails; id | email -----+-------------------- 1 | 2 | (2 rows) Time: 123.407227ms root@cockroachdb-02:26257/contacts> \q

Data has been replicated to second node, it shows that our CockroachDB cluster has been configured successfully.


Accessing CockroachDB Web Admin UI:

To access CockroachDB Web Admin UI, we require a user account. This user must be created within CockroachDB database.

Therefore, connect to SQL shell of any CockroachDB node and create a database user.

root@cockroachdb-01:26257/defaultdb> CREATE USER ahmer WITH PASSWORD 'cockroach'; CREATE ROLE Time: 363.507317ms

You are also required to grant the admin privilege to ahmer user.

root@cockroachdb-01:26257/defaultdb> GRANT admin TO ahmer; GRANT Time: 1.425956943s

Open URL https://cockroachdb-01:8080 in a web browser.


Login as ahmer user.




Configure Load Balancer for CockroachDB Cluster:

Each node in CockroachDB cluster has its own SQL and Web Admin UI services, that can be access independently by using Node hostname or IP address.

Therefore, it is necessary to configure a reverse proxy load balancer, so the users/applications can access our cluster by a common address/port instead of accessing individual nodes with separate hostname/IP addresses.

For this purpose, we can use HAProxy to configure a software load balancer. We can even generate a HAProxy configuration file by using cockroach command.

[root@cockroachdb-01 ~]# cockroach gen haproxy --certs-dir=/opt/cockroachdb/certs --host=cockroachdb-01

Check the content of haproxy.cfg file.

[root@cockroachdb-01 ~]# cat haproxy.cfg global maxconn 4096 defaults mode tcp # Timeout values should be configured for your specific use. # See: timeout connect 10s timeout client 1m timeout server 1m # TCP keep-alive on client side. Server already enables them. option clitcpka listen psql bind :26257 mode tcp balance roundrobin option httpchk GET /health?ready=1 server cockroach1 $(hostname):26257 check port 8080

This file requires minor adjustments and then it will perfectly work on a HAProxy load balancer.

You can refer to our previous post to configure HAProxy load balancer.



We have successfully installed and configured CockroachDB Secure Cluster in CentOS 8. If you having difficulty understanding the steps in this article, then you should buy and read RHCSA Red Hat Enterprise Linux 8: Training and Exam Preparation Guide (EX200), First Edition by Asghar Ghori.

No comments:

Post a Comment