Strategies for Generating Unique Identifiers in Sharded Database Systems

Unique identifier generation in distributed database environments solves critical challenges including shard key collisions, auto-increment sequence limitations, and cross-node uniqueness requirements. Effective solutions must produce globally unique values with optional ordering characteristics while maintaining high throughput performance.

Comparative Analysis of Identifier Generation Techniques

Method Core Mechanism Advantages Limitations Optimal Use Cases
Timestamp-Node Hybrid 64-bit integer: timestamp (41b) + node ID (10b) + sequence (12b) Local generation, high throughput, natural ordering, minimal dependencies Vulnerable to clock synchronization issues causing duplicates Primary transaction tables (orders, accounts, transactions)
Preallocated Ranges Business-specific ID blocks assigned via central allocator Straightforward implementation, sequential values, audit-friendly Database bottleneck, complex scaling, limited throughput Medium-scale applications with moderate write volumes
UUID Variants 128-bit values combining hardware identifiers and randomness Zero dependencies, infinite capacity, simple integration Poor indexing performance, random distribution, storage overhead Non-critical data (audit logs, session stores)
Centralized Sequence Single source with partitioned increment steps Familiar pattern, maintains ordering, easy migration path Single point of failure, scaling complexity, throughput ceiling Legacy system modernization projects
Buffered Allocation Local caching of pre-reserved ID blocks from service High throughput, ordering preservation, elastic scaling Middleware dependency, implementation complexity High-volume transaction systems (payments, inventory)
Atomic Counter Increment operations on distributed cache Sub-millisecond latency, guaranteed uniqueness, simple API Cache dependency, persistence requirements, unordered output High-frequency counters (inventory, voting systems)

Practical Implementation Patterns

Timestamp-Node Hybrid Generator (Recommended)

This approach creates 64-bit numeric identifiers structured as:

0 (sign) | 41-bit timestamp | 10-bit node identifier | 12-bit sequence counter
  • Timestamp base: 41 bits supports 69 years of millisecond precision
  • Node capacity: 10 bits enables 1,024 unique node identifiers
  • Concurrency handling: 12-bit counter allows 4,096 IDs per millisecond

Java implementation with clock drift protection:

public class TimestampIdProvider {
    private static final long EPOCH = 1704067200000L; // 2024-01-01 epoch
    private static final int NODE_BITS = 10;
    private static final int COUNTER_BITS = 12;
    
    private final long maxNodeValue = (1L << NODE_BITS) - 1;
    private final long maxCounterValue = (1L << COUNTER_BITS) - 1;
    private final int nodeShift = COUNTER_BITS;
    private final int timestampShift = COUNTER_BITS + NODE_BITS;
    
    private final long nodeId;
    private long lastTimestamp = -1L;
    private long sequenceCounter = 0L;

    public TimestampIdProvider(long nodeId) {
        if (nodeId > maxNodeValue || nodeId < 0) {
            throw new IllegalArgumentException("Invalid node identifier");
        }
        this.nodeId = nodeId;
    }

    public synchronized long nextIdentifier() {
        long currentTimestamp = System.currentTimeMillis();
        
        if (currentTimestamp < lastTimestamp) {
            throw new IllegalStateException("System clock regression detected");
        }
        
        if (currentTimestamp == lastTimestamp) {
            sequenceCounter = (sequenceCounter + 1) & maxCounterValue;
            if (sequenceCounter == 0) {
                currentTimestamp = waitForNextTimestamp();
            }
        } else {
            sequenceCounter = 0;
        }
        
        lastTimestamp = currentTimestamp;
        return ((currentTimestamp - EPOCH) << timestampShift)
               | (nodeId << nodeShift)
               | sequenceCounter;
    }

    private long waitForNextTimestamp() {
        long timestamp = System.currentTimeMillis();
        while (timestamp <= lastTimestamp) {
            timestamp = System.currentTimeMillis();
        }
        return timestamp;
    }
}

Database configuration:

  • Use BIGINT UNSIGNED column type
  • Establish primary key index on identifier column

Preallocated Range Allocator

Centralized allocation using database-managed blocks:

-- Identifier allocation table
CREATE TABLE id_allocation (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    service_key VARCHAR(32) NOT NULL UNIQUE,
    current_max BIGINT UNSIGNED NOT NULL DEFAULT 0,
    allocation_size INT NOT NULL DEFAULT 10000,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB;

-- Allocation procedure
DELIMITER $$
CREATE PROCEDURE acquire_id_block(
    IN service VARCHAR(32),
    OUT block_start BIGINT,
    OUT block_end BIGINT
)
BEGIN
    START TRANSACTION;
    UPDATE id_allocation 
    SET current_max = current_max + allocation_size
    WHERE service_key = service;
    
    SELECT current_max - allocation_size + 1, current_max
    INTO block_start, block_end
    FROM id_allocation
    WHERE service_key = service;
    COMMIT;
END$$
DELIMITER ;

Application workflow:

  1. Request block via acquire_id_block('orders', @start, @end)
  2. Locally increment from @start to @end
  3. Request new block when local counter reaches @end

Atomic Counter Service (Redis)

Leveraging Redis for high-frequency counters:

// Redis-backed identifier generator
public class RedisCounterProvider {
    private final Jedis redisClient;
    private final String counterKey;

    public RedisCounterProvider(Jedis client, String key) {
        this.redisClient = client;
        this.counterKey = key;
    }

    public long nextValue() {
        return redisClient.incr(counterKey);
    }

    public void initializeCounter(long initialValue) {
        redisClient.set(counterKey, String.valueOf(initialValue));
    }
}
  • Requires Redis persistence configuration
  • Ideal for temporary counters with millisecond-scale resolution
  • Not suitable for ordered primary keys

Implementation Guidance

Select timestamp-node hybrids for:

  • Transaction-critical systems requiring natural ordering
  • Sharded database architectures with high write volumes
  • Applications needing numeric primary keys

Opt for preallocated ranges when:

  • Operational simplicity outweighs performance needs
  • Existing MySQL expertise dominates the stack
  • Manual identifier adjustment is occasionally required

Reserve UUID variants for:

  • Non-indexed operational data stores
  • Decentralized microservice environments
  • Scenarios where identifier structure is irrelevant

Employ atomic counters specifically for:

  • Real-time inventory management
  • High-frequency voting or polling systems
  • Temporary state tracking with short retention

Critical considerations:

  • Implement clock synchronization safeguards for timestamp-based systems
  • Use transactional updates for allocation table modifications
  • Standardize on unsigned 64-bit integers across storage layers
  • Avoid over-engineering for applications under 1K writes/second

Tags: MySQL Sharding Snowflake Algorithm Database Segmentation Redis INCR UUID Generation

Posted on Tue, 29 Sep 2026 16:08:56 +0000 by lostcarpark