Virtual Machine Specifications
Using Alibaba Cloud instance type: ecs.u1-c1m4.4xlarge, PL2: Single disk IOPS performance limit 100,000 (applicable cloud disk capacity range: 461GiB - 64TiB)
| vCPU | Memory (GiB) | Disk (GB) | Bandwidth (Gbit/s) | Quantity |
|---|---|---|---|---|
| 16 | 64 | 500 | 5000 | 24 |
Easysearch Configuration
7-node cluster, version: 1.9.0
| Instance Name | Internal IP | Software | vCPU | JVM | Disk |
|---|---|---|---|---|---|
| search-node-01 | 172.22.75.144 | Easysearch | 16 | 31G | 500GB |
| search-node-02 | 172.23.15.97 | Easysearch | 16 | 31G | 500GB |
| search-node-03 | 172.25.230.228 | Easysearch | 16 | 31G | 500GB |
| search-node-04 | 172.22.75.142 | Easysearch | 16 | 31G | 500GB |
| search-node-05 | 172.22.75.143 | Easysearch | 16 | 31G | 500GB |
| search-node-06 | 172.24.250.252 | Easysearch | 16 | 31G | 500GB |
| search-node-07 | 172.24.250.254 | Easysearch | 16 | 31G | 500GB |
Elasticsearch Configuration
7-node cluster, version: 7.10.2
| Instance Name | Internal IP | Software | vCPU | JVM | Disk |
|---|---|---|---|---|---|
| es-node-01 | 172.24.250.251 | Elasticsearch | 16 | 31G | 500GB |
| es-node-02 | 172.22.75.145 | Elasticsearch | 16 | 31G | 500GB |
| es-node-03 | 172.17.67.246 | Elasticsearch | 16 | 31G | 500GB |
| es-node-04 | 172.22.75.139 | Elasticsearch | 16 | 31G | 500GB |
| es-node-05 | 172.22.75.140 | Elasticsearch | 16 | 31G | 500GB |
| es-node-06 | 172.24.250.253 | Elasticsearch | 16 | 31G | 500GB |
| es-node-07 | 172.24.250.250 | Elasticsearch | 16 | 31G | 500GB |
Monitoring Cluster Configuration
Single-node Easysearch cluster, version: 1.9.0
| Instance Name | Internal IP | Software | vCPU | Memory | Disk |
|---|---|---|---|---|---|
| monitor-node-01 | 172.25.230.226 | Monitoring: Console | 16 | 64G | 500GB |
| monitor-node-02 | 172.23.15.98 | Monitoring: Easysearch | 16 | 64G | 500GB |
Load Generator Configuration
loadgen version: 1.25.0
4 machines for Easysearch, 4 machines for Elasticsearch
| Instance Name | Internal IP | Software | vCPU | Memory | Disk |
|---|---|---|---|---|---|
| loadgen-es-01 | 172.17.67.245 | Loadgen - Easysearch | 16 | 64G | 500GB |
| loadgen-es-02 | 172.22.75.141 | Loadgen - Easysearch | 16 | 64G | 500GB |
| loadgen-es-03 | 172.25.230.227 | Loadgen - Easysearch | 16 | 64G | 500GB |
| loadgen-es-04 | 172.22.75.138 | Loadgen - Easysearch | 16 | 64G | 500GB |
| loadgen-el-01 | 172.24.250.255 | Loadgen - Elasticsearch | 16 | 64G | 500GB |
| loadgen-el-02 | 172.24.251.0 | Loadgen - Elasticsearch | 16 | 64G | 500GB |
| loadgen-el-03 | 172.24.250.248 | Loadgen - Elasticsearch | 16 | 64G | 500GB |
| loadgen-el-04 | 172.24.250.249 | Loadgen - Elasticsearch | 16 | 64G | 500GB |
Test Index Mapping
PUT web_logs
{
"mappings": {
"properties": {
"http_method": {
"type": "keyword"
},
"data_rate": {
"type": "integer"
},
"service": {
"type": "keyword"
},
"client_ip": {
"type": "ip"
},
"memory_consumption": {
"type": "integer"
},
"upstream_duration": {
"type": "float"
},
"request_uri": {
"type": "keyword"
},
"response_length": {
"type": "integer"
},
"request_duration": {
"type": "float"
},
"request_payload_size": {
"type": "integer"
},
"error_status": {
"type": "keyword"
},
"system_metrics": {
"properties": {
"queue_length": {
"type": "integer"
},
"memory_allocated": {
"type": "integer"
},
"thread_quantity": {
"type": "integer"
},
"processor_utilization": {
"type": "integer"
},
"active_sessions": {
"type": "integer"
}
}
},
"cpu_load": {
"type": "integer"
},
"user_agent_string": {
"type": "keyword"
},
"session_count": {
"type": "integer"
},
"event_time": {
"type": "date",
"format": "yyyy-MM-dd'T'HH:mm:ss.SSS"
},
"response_code": {
"type": "integer"
}
}
},
"settings": {
"number_of_shards": 7,
"number_of_replicas": 0,
"refresh_interval": "30s"
}
}
Testing Methodology
Every 4 load generators use the bulk write interface to stress test the same cluster's 7 nodes, with each request writing 10,000 documents.
Specific request configuration:
requests:
- request: #prepare test documents
method: POST
runtime_variables:
# batch_identifier: uuid
runtime_body_line_variables:
# routing_identifier: uuid
# url: $[[env.ES_ENDPOINT]]/_bulk
url: $[[ip]]/_bulk
body_repeat_times: 10000
basic_auth:
username: "$[[env.ES_USERNAME]]"
password: "$[[env.ES_PASSWORD]]"
body: |
{"index": {"_index": "web_logs", "_type": "_doc", "_id": "$[[uuid]]"}}
$[[message]]
Test Data Sample
{"http_method":"DELETE","data_rate":1955,"service":"cart-service","client_ip":"120.204.26.240","memory_consumption":1463,"upstream_duration":"1.418","request_uri":"/health","response_length":421,"request_duration":"0.503","request_payload_size":1737,"error_status":"SYSTEM_ERROR","system_metrics":{"queue_length":769,"memory_allocated":1183,"thread_quantity":65,"processor_utilization":68,"active_sessions":837},"cpu_load":70,"user_agent_string":"Mozilla/5.0 (iPad; CPU OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1","session_count":54,"event_time":"2024-11-16T14:25:21.423","response_code":500}
{"http_method":"OPTIONS","data_rate":10761,"service":"product-service","client_ip":"223.99.83.60","memory_consumption":567,"upstream_duration":"0.907","request_uri":"/static/js/app.js","response_length":679,"request_duration":"1.287","request_payload_size":1233,"error_status":"NOT_FOUND","system_metrics":{"queue_length":565,"memory_allocated":1440,"thread_quantity":148,"processor_utilization":39,"active_sessions":1591},"cpu_load":87,"user_agent_string":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1","session_count":354,"event_time":"2024-11-16T05:37:28.423","response_code":502}
{"http_method":"HEAD","data_rate":10257,"service":"recommendation-service","client_ip":"183.60.242.143","memory_consumption":1244,"upstream_duration":"0.194","request_uri":"/api/v1/recommendations","response_length":427,"request_duration":"1.449","request_payload_size":1536,"error_status":"UNAUTHORIZED","system_metrics":{"queue_length":848,"memory_allocated":866,"thread_quantity":86,"processor_utilization":29,"active_sessions":3846},"cpu_load":71,"user_agent_string":"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","session_count":500,"event_time":"2024-11-16T15:14:30.424","response_code":403}
Test Results: 1 Primary Shard, 0 Replicas
Elasticsearch Throughput
Elasticsearch Thread and Queue Performance
Resource Consumption
Easysearch Throughput
Easysearch Thread and Queue Performance
Resource Consumption
Comparison
| Software | Average Cluster Throughput | Average Single Node Throughput | Maximum Queue | Disk Consumption |
|---|---|---|---|---|
| Elasticsearch | 50,000 ops/sec | 50,000 ops/sec | 811 | 10GB |
| Easysearch | 70,000 ops/sec | 70,000 ops/sec | 427 | 4GB |
Test Results: 1 Primary Shard, 1 Replica
Elasticsearch Throughput
Elasticsearch Thread and Queue Performance
Resource Consumption
Easysearch Throughput
Easysearch Thread and Queue Performance
Resource Consumption
Comparison
| Software | Average Cluster Throughput | Average Single Node Throughput | Maximum Queue | Disk Consumption (~30 million documents) |
|---|---|---|---|---|
| Elasticsearch | 100,000 ops/sec | 50,000 ops/sec | 791 | 22GB |
| Easysearch | 140,000 ops/sec | 70,000 ops/sec | 421 | 7GB |
Test Results: 7 Primary Shards
Elasticsearch Throughput
Elasticsearch Thread and Queue Performance
Resource Consumption
Network
Single node average reception: 26MB/s, corresponding bandwidth: 1456 Mb/s
50 million documents, total storage: 105 GB, single node: 15 GB
Easysearch Throughput
Easysearch Thread and Queue Performance
Resource Consumption
Comparison
| Software | Average Cluster Throughput | Average Single Node Throughput | Maximum Queue | Disk Consumption |
|---|---|---|---|---|
| Elasticsearch | 350,000 ops/sec | 50,000 ops/sec | 2449 | 105GB |
| Easysearch | 600,000 ops/sec | 85,714 ops/sec | 1172 | 36GB |
Summary
Based on comparative analysis of test results across different scenarios, the following conclusions can be drawn:
-
Easysearch demonstrates significantly improved indexing performance compared to Elasticsearch Easysearch cluster throughput performance improved by 40% - 70%, and the performance improvement effect becomes more significant as the number of shards increases.
-
Easysearch shows substantially better disk compression efficiency compared to Elasticsearch Easysearch cluster disk compression efficiency improved by 2.5 - 3 times, and the compression effect becomes more apparent as data volume increases.
These test results indicate that Easysearch offers superior performance and storage efficiency advantages in log processing scenarios, particularly suitable for large-scale shard and massive data usage scenarios.