Prerequisites

Before configuring replication, ensure the following conditions are met on both the source and target clusters:

• Both clusters must have the cross-cluster-replication and index-management plugins installed.
• If the easysearch.yml configuration file on the target cluster defines custom node.roles, it must expilcitly include the remote_cluster_client role to enable outgoing connections.

Demonstration Environment

• Source Cluster (Leader): 192.168.3.45:9200
• Target Cluster (Follower): 192.168.3.39:9200
• Security features (SSL/TLS) are enabled on both nodes.

Establishing Mutual Certificate Trust

To facilitate secure communication, the Certificate Authority (CA) certificates from both clusters must be combined into a single trust chain file.

cat source-ca.crt target-ca.crt > shared-truststore.pem

Update the easysearch.yml file on both nodes to reference this combined file instead of the individual CA certificate.

#security.ssl.transport.ca_file: ca.crt
security.ssl.transport.ca_file: shared-truststore.pem

Configuring Remote Connectivity

On the target cluster, define the connection settings for the source cluster. This involves setting up the cluster alias and seed hosts.

PUT /_cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "prod_source": {
          "seeds": ["192.168.3.45:9300"]
        }
      }
    }
  }
}

Initiating Manual Replication

First, populate the source cluster with sample data. If an index already exists, this step can be skipped.

POST /_bulk
{ "index" : { "_index" : "app_data", "_id" : "101" } }
{ "cpu_usage" : "45%", "status" : "active" }
{ "create" : { "_index" : "app_data", "_id" : "102" } }
{ "cpu_usage" : "12%", "status" : "idle" }

On the target cluster, initiate the replication process. This command creates a local index named app_data_replica that synchronizes with the app_data index on the source.

PUT /_replication/app_data_replica/_start
{
   "leader_alias": "prod_source",
   "leader_index": "app_data",
   "use_roles":{
      "leader_cluster_role": "cross_cluster_replication_leader_full_access",
      "follower_cluster_role": "cross_cluster_replication_follower_full_access"
   }
}

Verifying Replication Status

To confirm that the replication is active and synchronized, query the status API on the target cluster.

GET /_replication/app_data_replica/_status

To test real-time synchronization, insert new documents into the source cluster.

POST /_bulk
{ "index" : { "_index" : "app_data", "_id" : "103" } }
{ "cpu_usage" : "88%", "status" : "warning" }
{ "create" : { "_index" : "app_data", "_id" : "104" } }
{ "cpu_usage" : "5%", "status" : "idle" }

Querying the app_data_replica index on the target cluster should now return documents 103 and 104.

Managing Replication: Pause and Resume

Maintenance operations may require temporarily halting the replication process without removing the follower index configuration.

Pausing Replication

Use the pause command to stop the data fetch. Changes on the leader index will not be synchronized during this time.

POST /_replication/app_data_replica/_pause
{}

Checking the status at this point should return a state of PAUSED.

GET /_replication/app_data_replica/_status

Resuming Replication

To restart the synchronization process, issue the resume command.

POST /_replication/app_data_replica/_resume
{}

Stopping Replication

Follower indices are strictly read-only to maintain data consistency. To convert the follower index into a standard, writable index, the replication must be stopped. Attempting to write data before stopping will result in an error.

Execute the stop command to decouple the index. Once stopped, the index operates independently and will no longer receive updates from the source.

POST /_replication/app_data_replica/_stop
{}

After this operation, writing to the index on the target cluster will succeed.

Configuring Auto-Follow Patterns

For environments utilizing time-series indices (e.g., daily log indices), manually starting replication for each new index is inefficient. The auto-follow feature allows the target cluster to automatically replicate any new index created on the source that matches a specific pattern.

Creating an Auto-Follow Pattern

Define a replication policy on the target cluster. The following example sets up a rule to automatically replicate any index starting with audit-logs from the source cluster.

POST /_replication/_autofollow
{
   "leader_alias" : "prod_source",
   "name": "audit-log-policy",
   "pattern": "audit-logs*",
   "use_roles":{
      "leader_cluster_role": "cross_cluster_replication_leader_full_access",
      "follower_cluster_role": "cross_cluster_replication_follower_full_access"
   }
}

Create indices matching this pattern on the source cluster:

PUT /audit-logs-2023-01/_doc/1
{ "level": "info", "msg": "User login" }

PUT /audit-logs-2023-02/_doc/1
{ "level": "error", "msg": "Connection timeout" }

The target cluster will automatically create corresponding follower indices for these.

Deleting Auto-Follow Patterns

To stop the automatic replication of future indices, delete the auto-follow pattern. Note that this action does not affect existing follower indices that have already been created; they will continue replicating until individually stopped.

DELETE /_replication/_autofollow
{
   "leader_alias" : "prod_source",
   "name": "audit-log-policy"
}

To make existing replicated indices like audit-logs-2023-01 writable, you must explicitly stop replication on them using the _stop API.

Summary

This guide demonstrated the lifecycle of cross-cluster replication in Easysearch, covering the establishment of certificate trust, configuration of remote connections, and the management of both manual and pattern-based replication tasks. These capabilities provide a robust foundation for data redundancy, load distribution, and latency optimization in distributed search architectures.