In search engine architectures, compression algorithm selection presents a fundamental tradeoff dilemma:
- High compression ratios (e.g.,
best_compression/DEFLATE) reduce storage but slow query decompression - Fast encoding schemes (e.g., default LZ4) accelerate queries but increase disk footprint
Easysearch introduces a solution leveraging JDK 21 FFM (Foreign Function & Memory API) to directly interface with native ZSTD lirbaries. We conducted a controlled benchmark comparing Easysearch (ZSTD) and Elasticsearch 7.10.2 (best_compression) under identical conditions.
Results demonstrate significant performance gains: Even under substantial system load, Easysearch achieves nearly 5x higher query throughput while maintaining high compression.
Test Environment
Identical configurations ensured fair comparison:
| Configuration | Easysearch | Elasticsearch 7.10.2 |
|---|---|---|
| Nodes | 3 | 3 |
| JVM Heap | 12GB × 3 | 12GB × 3 |
| node.procesors | 16 × 3 | 16 × 3 |
| Documents | 10,000,000 | 10,000,000 |
| Primary/Replica Shards | 3 / 0 | 3 / 0 |
| Data Type | nginx access logs | nginx access logs |
| Fields | 17 | 17 |
| Stored Fields Compression | ZSTD (JDK21 FFM/native, level=3) | best_compression (DEFLATE) |
Query model: JMeter random match queries against service_name, method, error_code, and url fields, returning 10 documents per request.
Performance Results
1. Query Throughput (QPS): 372% Higher Under Load
| Metric | Elasticsearch (DEFLATE) | Easysearch (ZSTD) | Difference |
|---|---|---|---|
| Steady-State QPS | 532.8 | 2,518.0 | +372.6% |
| Avg Response Time | 779.0 ms | 164.3 ms | -78.9% |
2. CPU Efficiency: 4.88x Higher Throughput per CPU Unit
| Metric | Elasticsearch | Easysearch | Multiplier |
|---|---|---|---|
| QPS per 1% CPU | 5.76 | 28.10 | 4.88× |
3. Storage Footprint
| Index | Compression | Disk Usage |
|---|---|---|
| nginx_best_10m (ES) | best_compression | 1.8 GB |
| nginx_zstd3 (Easysearch) | ZSTD level=3 | 1.9 GB |
Technical Explanation
ZSTD's asymmetric compression design enables high ratios without compromising decompression speed. For stored fields (_source) access:
- DEFLATE becomes a CPU bottleneck during decompression
- ZSTD's native implementation reduces decompression overhead significantly
This efficiency allows Easysearch to process more concurrent queries using fewer CPU resources.
Implementation Guide
1. Enable ZSTD for New Indices
curl -k -u 'admin:securepass' -X PUT 'https://localhost:9200/new_index' \\
-H 'Content-Type: application/json' -d '{
"settings": {
"index.codec": "ZSTD",
"index.compression.zstd.jni": true
}
}'
2. Migrate Existing Indices
# Create target index
curl -k -u 'admin:securepass' -X PUT 'https://localhost:9200/target_index' \\
-H 'Content-Type: application/json' -d '{
"settings": {
"index.codec": "ZSTD",
"index.compression.zstd.jni": true
}
}'
# Reindex data
curl -k -u 'admin:securepass' -X POST 'https://localhost:9200/_reindex' \\
-H 'Content-Type: application/json' -d '{
"source": { "index": "source_index" },
"dest": { "index": "target_index" }
}'
3. Verify Configuration
curl -k -u 'admin:securepass' \\
'https://localhost:9200/index_name/_settings?include_defaults=true&pretty'
Confirm index.codec = ZSTD and index.compression.zstd.jni = true