Cluster Topology Requirements

To ensure high availability, an Elasticsearch cluster requires a minimum of three master-eligible nodes. This configuration allows the cluster to sustain the failure of a single master node without losing quorum. If two master nodes fail simultaneous, the cluster cannot form a quorum, resulting in service interruption and connection failures for clients like Kibana.

In this guide, we will simulate a multi-node cluster environment on a single virtual machine. Since running multiple instances concurrently consumes significant resources, allocate at least 3GB of RAM and 4 vCPUs to the host VM.

1. Environment Preparation and Instance Copying

We begin by deploying three distinct Elasticsearch instances from a single installation package. This approach creates a pseudo-cluster suitable for development and testing.

Execute the following commands to duplicate the base installation directory:

cd /opt
cp -r elasticsearch-7.4.0 es-node-01
cp -r elasticsearch-7.4.0 es-node-02
cp -r elasticsearch-7.4.0 es-node-03

Below is the planned network configuration for our nodes:

2. Directory and Permission Setup

Create dedicated directories for data storage and logging, then assign ownership to the non-root user running the service (e.g., esuser).

cd /opt
mkdir -p data logs
chown -R esuser:esuser ./data
chown -R esuser:esuser ./logs
chown -R esuser:esuser ./es-node-01
chown -R esuser:esuser ./es-node-02
chown -R esuser:esuser ./es-node-03

3. Configuring Elasticsearch Nodes

Edit the elasticsearch.yml file for each node. The configuration ensures unique identification while maintaining cluster cohesion.

Node 1 Configuration

Edit /opt/es-node-01/config/elasticsearch.yml:

cluster.name: production-es
node.name: node-01
node.master: true
node.data: true
node.max_local_storage_nodes: 3
network.host: 0.0.0.0
http.port: 9201
transport.tcp.port: 9301
discovery.seed_hosts: ["127.0.0.1:9301", "127.0.0.1:9302", "127.0.0.1:9303"]
cluster.initial_master_nodes: ["node-01", "node-02", "node-03"]
path.data: /opt/data
path.logs: /opt/logs

Node 2 Configuration

Edit /opt/es-node-02/config/elasticsearch.yml:

cluster.name: production-es
node.name: node-02
node.master: true
node.data: true
node.max_local_storage_nodes: 3
network.host: 0.0.0.0
http.port: 9202
transport.tcp.port: 9302
discovery.seed_hosts: ["127.0.0.1:9301", "127.0.0.1:9302", "127.0.0.1:9303"]
cluster.initial_master_nodes: ["node-01", "node-02", "node-03"]
path.data: /opt/data
path.logs: /opt/logs

Node 3 Configuration

Edit /opt/es-node-03/config/elasticsearch.yml:

cluster.name: production-es
node.name: node-03
node.master: true
node.data: true
node.max_local_storage_nodes: 3
network.host: 0.0.0.0
http.port: 9203
transport.tcp.port: 9303
discovery.seed_hosts: ["127.0.0.1:9301", "127.0.0.1:9302", "127.0.0.1:9303"]
cluster.initial_master_nodes: ["node-01", "node-02", "node-03"]
path.data: /opt/data
path.logs: /opt/logs

4. JVM Memory Tuning

Given the resource constraints of a single VM, reduce the default heap size allocated to the JVM. Modify the jvm.options file in the config directory of each node.

# Change default settings to 256MB
-Xms256m
-Xmx256m

5. Starting the Cluster

Launch the Elasticsearch instances sequentially. It is recommended to switch to the dedicated user (e.g., esuser) before starting.

# Start Node 1
/opt/es-node-01/bin/elasticsearch -d

# Start Node 2
/opt/es-node-02/bin/elasticsearch -d

# Start Node 3
/opt/es-node-03/bin/elasticsearch -d

During startup, logs may indicate "master not discovered yet." This is normal until the quorum is formed. Once the second node joins, a master election occurs, and the cluster status changes to active.

6. Verifying Cluster Status

Use the cat API to check the health of the cluster:

curl http://192.168.1.100:9201/_cat/health?v

The output provides several key metrics:

• status: The health indicator. green signifies optimal health; yellow means all primary shards are assigned but replicas are missing; red indicates some primary shards are unassigned.
• node.total: The number of active nodes in the cluster (should be 3).
• node.data: The count of nodes storing data.
• shards / pri: Total shards and primary shards count.
• active_shards_percent: Percentage of active shards (should be 100).

7. Configuring Kibana Integration

To visualize and manage the cluster, configure a separate instance of Kibana.

Copy and Configure Kibana

cd /opt
cp -r kibana-7.4.0-linux-x86_64 kibana-cluster

Update the configuration file kibana-cluster/config/kibana.yml to point to all Elasticsearch nodes:

elasticsearch.hosts: ["http://localhost:9201", "http://localhost:9202", "http://localhost:9203"]

Start Kibana:

./kibana-cluster/bin/kibana --allow-root

Monitoring the Cluster

Access the Kibana web interface and navigate to the "Stack Monitoring" section. Here, you can view the overal cluster health, which should display a green status. The "Nodes" tab will list the three configured nodes, with a star icon denoting the currently elected master node.