Standalone Server Limitations

Relying on a single Elasticsearch instance introduces several critical constraints that hinder scalability and resilience:

• Storage capacity boundaries on a single machine.
• Single point of failure risks, preventing high availability.
• Restricted concurrent request throughput.

Deploying a distributed cluster architecture mitigates these vulnerabilities.

Data Partitioning via Sharding

To overcome storage limits on a single machine, datasets are horizontally split into segments called shards. Each shard is hosted on a distinct node, distributing the storage burden across the cluster.

Cross-Node Replication

While sharding resolves capacity issues, a node failure would render a portion of the dataset inaccessible. Replication addresses this by creating copies (replicas) of each shard on different nodes.

To balance hardware costs with high availability, shards are distributed across nodes, and replicas are placed on peer nodes. For instance, with three primary shards and one replica per shard, the cluster remains fully functional even if one node goes offline, as the missing data is retrievable from its replica. Additionally, query load is distributed across these nodes, enhancing concurrent processing power.

Deploying a Multi-Node Cluster

A three-node cluster can be simulated on a single host by assigning unique ports. The target cluster name is prod-cluster, with the following node specifications:

• es-node-a: HTTP 9210, Transport 9310
• es-node-b: HTTP 9211, Transport 9311
• es-node-c: HTTP 9212, Transport 9312

Ensure all operations are executed using a non-root user (e.g., esadmin). Creating a system snapshot before proceeding is highly recommended.

1. Purging Existing Data

Stop the running instance and remove the existing data directory to prevent conflicts:

rm -rf /var/lib/elasticsearch/data

2. Adjusting Node A Configuration

Navigate to the configuration directory and edit elasticsearch.yml:

cluster.name: prod-cluster
node.name: es-node-a
path.data: /opt/es-cluster/node-a/data
path.logs: /opt/es-cluster/node-a/logs
network.host: 0.0.0.0
http.port: 9210
transport.tcp.port: 9310
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9310", "127.0.0.1:9311", "127.0.0.1:9312"]
discovery.zen.minimum_master_nodes: 2

3. Cloning Instances

Rename the original directory and replicate it for the remaining nodes:

mv elasticsearch node-a
cp node-a node-b -R
cp node-a node-c -R

Update node-b/config/elasticsearch.yml with unique values:

node.name: es-node-b
path.data: /opt/es-cluster/node-b/data
path.logs: /opt/es-cluster/node-b/logs
http.port: 9211
transport.tcp.port: 9311

Apply similar changes to node-c, incrementing the identifiers and ports accordingly.

4. Initializing the Cluster

Launch the instances in daemon mode:

./node-a/bin/elasticsearch -d
./node-b/bin/elasticsearch -d
./node-c/bin/elasticsearch -d

5. Common Startup Failures

Node Conflict Error: Cloning directories retains the data from Node A, causing uniqueness conflicts. Clear the data folder inside node-b and node-c before starting them.

Memory Allocation Error: The default JVM heap size (1GB) might exceed available memory on a single host. Modify the jvm.options file to reduce the allocation (e.g., -Xms256m -Xmx256m), with 128MB as the absolute minimum.

Cluster Health Indicators

Cluster status can be monitored via management tools, characterized by three states:

• Green: All primary and replica shards are active. The cluster is fully operational.
• Yellow: All primary shards are active, but one or more replicas are unassigned. Data is intact, but redundancy is compromised. This serves as a warning.
• Red: One or more primary shards are missing. Data queries will be incomplete, and write operations targeting the missing shards will fail.

Provisioning Indices in a Clustered Environment

When defining an index in a cluster, shard and replica allocation determines data distribution.

Index creation API structure:

PUT /product_catalog
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  }
}

Configuration parameters:

• number_of_shards: Defines the number of primary shards (3 in this case).
• number_of_replicas: Defines the number of copies per primary shard (1 here, resulting in 2 total copies per shard).

Through a monitoring interface, the shard layout for product_catalog might appear as follows:

• es-node-a holds primary shard 0 and replica of shard 2.
• es-node-b holds primary shard 1 and replica of shard 0.
• es-node-c holds primary shard 2 and replica of shard 1.