Cluster Discovery Configuration Fundamentals

Setting up an Elasticsearch cluster correctly is critical for stability, particularly regarding how nodes discover one another and elect a master. In version 7.x, several configuration parameters underwent significant changes compared to previous releases. This guide clarifies the distinctions between seed hosts and initial master nodes, explains the voting mechanism, and outlines best practices for fault tolerance.

Core Terminology and Roles

To grasp the configuration requirements, one must understand the specific roles nodes play within the distributed system:

• Master-Eligible Nodes: Candidates capable of being elected as the cluster leader. They manage metadata and cluster state but typically do not store data unless explicitly configured otherwise.
• Data Nodes: Responsible for storing shards and executing data-heavy operations like indexing and querying.
• Coordinating Nodes: Act solely as request routers, balancing load without holding cluster state or data.
• Voting Configuration: A persistent record maintained by master-eligible nodes that tracks which nodes are allowed to participate in elections.

The primary goal of this architecture is to prevent split-brain scenarios where multiple nodes believe they are the active master simultaneously.

Configuration Parameters: 6.x vs 7.x

Elasticsearch 7.x introduced stricter rules to simplify cluster management and improve safety. Below is a comparison of the relevant settings:

Dissecting the Settings

discovery.seed_hosts: This list informs a node about potential peers to contact during startup. While legacy versions allowed non-master-eligible nodes here, version 7.x restricts this list primarily to master-eligible nodes to ensure robust discovery paths. This setting is checked every time a node starts.

cluster.initial_master_nodes: This parameter defines the set of nodes responsible for bootstrapping the cluster. It is required only during the very first initialization of the cluster. Once the cluster state is established and a master is elected, this setting becomes obsolete but does not cause errors if left defined. It should contain the node identities of all master-eligible nodes participating in the initial vote.

--- Example elasticsearch.yml for node-alpha ---
node.name: node-alpha
node.master: true
node.data: true
network.host: 192.168.1.10
discovery.seed_hosts: ["192.168.1.10", "192.168.1.11"]
cluster.initial_master_nodes: ["node-alpha", "node-beta", "node-gamma"]

The Discovery Protocol

The discovery process occurs in two distinct phases when a node initializes:

1. Seed Probing: The node attempts to connect to the addresses specified in discovery.seed_hosts. It identifies which of these connected nodes are master-eligible.
2. Peer Exchange: Upon finding eligible candidates, the node shares its known list of master candidates with them. These peers reciprocate, allowing the node to build a complete map of potential masters.

If a candidate cannot locate a leader or sufficient peers, it enters a retry loop. Crucially, once a stable cluster exists, new nodes join via the existing master rather than re-triggering a full bootstrap election, provided the new node recognizes the current cluster name and UUID.

Quorum and Fault Tolerance

Cluster health relies on achieving a quorum, defined as a majority of voting nodes (N/2 + 1). If more than half of the voting configuration becomes unavailable, the cluster freezes to prevent data inconsistency.

• Odd Numbers: It is standard practice to deploy an odd number of master-eligible nodes (e.g., 3, 5, or 7). This ensures that network partitions result in a clear majority decision.
• Even Numbers: If an even number is deployed, Elasticsearch effectively treats it as N-1 for quorum calculations, offering no additional benefit over the next lower odd number.

Managing Node Removal

Removing nodes requires caution. If you plan to shut down enough nodes that the remaining count falls below the quorum threshold, you must update the voting configuration beforehand.

Use the Voting Config Exclusions API to temporarily remove nodes from the election pool before shutting them down:

# Exclude specific nodes by name
POST /_cluster/voting_config_exclusions
{
  "node_names": [ "node-to-remove" ]
}

# Clear exclusions after nodes are safely removed
DELETE /_cluster/voting_config_exclusions

If removing fewer than half of the master-eligible nodes, this step is optional but recommended for a graceful transition.

Testing Failure Secnarios

Practical validation confirms the theoretical behaviors described above. Several test cases illustrate the resilience boundaries:

• Scenario: Single Master Failure. In a three-node cluster where all are master-eligible, terminating the current master triggers an immediate election. One of the surviving candidates assumes leadership, ensuring availability.
• Scenario: Majority Loss. In the same three-node setup, terminating two nodes results in cluster paralysis. The system halts operations because the single remaining node cannot achieve a majority quorum. Error logs indicate a failure to satisfy the election requirement.
• Scenario: Ignoring Bootstrap Settings. After a cluster has successfully initialized, removing the cluster.initial_master_nodes definition from the YAML file does not impact runtime behavior. The setting is read-only at bootstrap time.

Addressing Common Configuration Queries

Based on production experiences, here are answers to frequent questions regarding cluster topology:

Must seed hosts contain only master-eligible nodes? Ideally, yes. Using data-only nodes as seeds can introduce fragility during recovery.

Can we modify initial_master_nodes after startup? Technically, the system ignores changes to this value once the cluster is running, but it is cleaner to leave it unchanged or remove it entirely post-initialization.

Does removing nodes lower the quorum dynamically? No. The quorum requirement remains tied to the voting configuration size. If the original configuration expected 5 votes, removing 3 leaves 2, which is insufficient for a quorum of 5. However, if the exclusion API is used to formally remove them from the voting config, the new quorum requirement adjusts automatically based on the remaining active voters.

Do decommissioned nodes form a split brain upon restart? Generally, no. Nodes retain their cluster state and UUID. Even if they reconnect to each other in isolation, they recognize they are part of a larger cluster state that includes the online majority and will attempt to rejoin the active partition rather than forming a rogue sub-cluster.