In high-volume data ingestion scenarios, insufficiant capacity planning often leads to cluster instability. Common issues include:

• Excessive memory consumption causing Out-Of-Memory (OOM) errors and node disconnections.
• High CPU and memory usage degrading the performance of search queries.
• Increased disk I/O and network saturation affecting cluster health.

Previously, mitigating sudden traffic spikes required deploying external "buffer" services to intercept and throttle requests before they reached the database. Starting with version 1.8.0, Easysearch introduced native node and shard-level throttling, adding index-level controls in version 1.8.2. These features allow administrators to manage ingestion pressure directly within the cluster.

Note: Throttling limits apply to Primary Shards. Total throughput including replicas will be a multiple of the limit set.

Test Environment

The following setup was used to benchmark the throttling features:

ip       name   http          port version role master
10.0.0.3 node-3 10.0.0.3:9209 9303 1.8.0   dimr -
10.0.0.3 node-4 10.0.0.3:9210 9304 1.8.0   im   -
10.0.0.3 node-2 10.0.0.3:9208 9302 1.8.0   dimr -
10.0.0.3 node-1 10.0.0.3:9207 9301 1.8.0   dimr *

Index configuration used for testing:

PUT test_0
{
  "settings": {
    "number_of_replicas": 1,
    "number_of_shards": 3
  }
}

Stress testing was performed using INFINI Loadgen simulating 10 concurrent threads, sending bulks of 5000 documents for 180 seconds.

Node-Level Throttling

Without restrictions, a single data node processed approximately 40,000 docs/s with ~10% CPU usage. To enforce a hard limit, configure the cluster settings to restrict ingestion to 10,000 docs/s per node.

PUT _cluster/settings
{
  "transient": {
    "cluster.throttle.node.write": true,
    "cluster.throttle.node.write.max_requests": 10000,
    "cluster.throttle.node.write.action": "retry"
  }
}

After applying this limit, CPU usage dropped by roughly 50%. Accounting for replicas, the total throughput observed was 20,000 docs/s.

Shard-Level Throttling

To prevent a single busy shard from overwhelming a node, you can set limits per primary shard. The following configuration restricts each shard to 2,000 docs/s.

PUT _cluster/settings
{
  "transient": {
    "cluster.throttle.shard.write": true,
    "cluster.throttle.shard.write.max_requests": 2000,
    "cluster.throttle.shard.write.action": "retry"
  }
}

Monitoring the test_0 index shows the primary shard ingestion rate stabilizing at 6,000 docs/s (3 shards * 2,000 limit). Including replicas, the total shard activity on the node reached 4,000 docs/s.

Index-Level Throttling

If a specific index is consuming disproportionate resources, you can apply limits directly to that index's settings. Here is how to cap the test_0 index at 6,000 docs/s:

PUT test_0
{
  "settings": {
    "number_of_replicas": 1,
    "number_of_shards": 3,
    "index.throttle.write.enable": true,
    "index.throttle.write.max_requests": 6000,
    "index.throttle.write.action": "retry"
  }
}

Metrics confirm the primary indexing rate for the index remains capped at the 6,000 docs/s threshold.