System Architecture and Topology

This deployment utilizes Docker Swarm to orchestrate a highly available TDengine 3.3.6 cluster. The infrastructure consists of three physical or virtual nodes designated as server-node1, server-node2, and server-node3. The IP addressing scheme for this setup assumes a 10.0.0.x subnet.

Swarm Role Assignment

• Swarm Manager: server-node1 (10.0.0.1) handles orchestration and management.
• Swarm Workers: server-node2 (10.0.0.2) and server-node3 (10.0.0.3) execute container workloads.

Service Components

Component	Function	Placement
taos-dnode-01/02/03	Core TDengine data nodes	Global (1 per node)
taos-adapter	RESTful interface bridge	Global (3 replicas)
taos-keeper	Metrics collection and monitoring	Global (1 replica)
taos-explorer	Web-based management UI	Global (1 replica)
ingress-gateway	Nginx reverse proxy for load balancing	Global (2 replicas)

1. Environment Preparation

1.1 Hostname Resolution

Ensure mutual DNS resolution across all servers by appending the following to /etc/hosts on every node:

echo "10.0.0.1 server-node1" >> /etc/hosts
echo "10.0.0.2 server-node2" >> /etc/hosts
echo "10.0.0.3 server-node3" >> /etc/hosts

1.2 Initializing the Swarm Cluster

On server-node1, initialize the Swarm manager:

docker swarm init --advertise-addr 10.0.0.1

Join the worker nodes by executing the command generated by the manager on server-node2 and server-node3:

docker swarm join --token SWMTKN-1-xxxxxxxx 10.0.0.1:2377

Verify the cluster status on the manager:

docker node ls

1.3 Storage Provisioning

Create persistent directories for TDengine data and logs on each respective host. Adjust permissions to allow container write access.

# On server-node1
mkdir -p /srv/tddata/dn1/{lib,log}
chmod -R 777 /srv/tddata/dn1

# On server-node2
mkdir -p /srv/tddata/dn2/{lib,log}
chmod -R 777 /srv/tddata/dn2

# On server-node3
mkdir -p /srv/tddata/dn3/{lib,log}
chmod -R 777 /srv/tddata/dn3

2. Docker Stack Configuration

The deployment is split into stages to ensure correct dependency order. The internal overlay network is named taos-net.

2.1 Primary Data Node (Stage 1)

Deploy stack-phase-1.yml to start the first dnode.

version: '3.8'

networks:
  taos-net:
    driver: overlay
    attachable: true

services:
  dnode-01:
    image: tdengine/tdengine:3.3.6.13
    hostname: dnode-01
    networks:
      - taos-net
    volumes:
      - /srv/tddata/dn1/lib:/var/lib/taos
      - /srv/tddata/dn1/log:/var/log/taos
      - /srv/tddata/config/taos.cfg:/etc/taos/taos.cfg
      - /etc/localtime:/etc/localtime:ro
    environment:
      - TZ=Asia/Shanghai
    deploy:
      placement:
        constraints: [node.hostname == server-node1]
      restart_policy:
        condition: on-failure
    command: taosd

2.2 Secondary Data Nodes (Stage 2)

Deploy stack-phase-2.yml to add the remaining nodes.

version: '3.8'

networks:
  taos-net:
    external: true

services:
  dnode-02:
    image: tdengine/tdengine:3.3.6.13
    hostname: dnode-02
    networks:
      - taos-net
    volumes:
      - /srv/tddata/dn2/lib:/var/lib/taos
      - /srv/tddata/dn2/log:/var/log/taos
      - /srv/tddata/config/taos.cfg:/etc/taos/taos.cfg
      - /etc/localtime:/etc/localtime:ro
    deploy:
      placement:
        constraints: [node.hostname == server-node2]
    command: taosd

  dnode-03:
    image: tdengine/tdengine:3.3.6.13
    hostname: dnode-03
    networks:
      - taos-net
    volumes:
      - /srv/tddata/dn3/lib:/var/lib/taos
      - /srv/tddata/dn3/log:/var/log/taos
      - /srv/tddata/config/taos.cfg:/etc/taos/taos.cfg
      - /etc/localtime:/etc/localtime:ro
    deploy:
      placement:
        constraints: [node.hostname == server-node3]
    command: taosd

2.3 Supporting Components (Stage 3)

Deploy stack-phase-3.yml for adapter, keeper, and explorer.

version: '3.8'

networks:
  taos-net:
    external: true

services:
  adapter:
    image: tdengine/tdengine:3.3.6.13
    networks:
      - taos-net
    ports:
      - target: 6041
        published: 6041
        mode: host
    environment:
      - TAOS_FIRST_EP=dnode-01:6030
    volumes:
      - /srv/tddata/config/taosadapter.toml:/etc/taos/taosadapter.toml
      - /etc/localtime:/etc/localtime:ro
    deploy:
      mode: global
    command: taosadapter

  keeper:
    image: tdengine/tdengine:3.3.6.13
    networks:
      - taos-net
    environment:
      - TAOS_FIRST_EP=dnode-01:6030
    volumes:
      - /srv/tddata/config/taoskeeper.toml:/etc/taos/taoskeeper.toml
      - /etc/localtime:/etc/localtime:ro
    deploy:
      replicas: 1
      placement:
        constraints: [node.role == worker]
    command: taoskeeper

  explorer:
    image: tdengine/tdengine:3.3.6.13
    networks:
      - taos-net
    environment:
      - TAOS_FIRST_EP=dnode-01:6030
    volumes:
      - /srv/tddata/config/explorer.toml:/etc/taos/explorer.toml
      - /etc/localtime:/etc/localtime:ro
    deploy:
      replicas: 1
    command: taos-explorer

2.4 Ingress Gateway (Stage 4)

Deploy stack-phase-4.yml to configure Nginx.

version: '3.8'

networks:
  taos-net:
    external: true

services:
  gateway:
    image: nginx:alpine
    networks:
      - taos-net
    ports:
      - "6060:6060"
    volumes:
      - /srv/tddata/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - /etc/localtime:/etc/localtime:ro
    deploy:
      replicas: 2
      restart_policy:
        condition: any

3. Configuration Details

3.1 Nginx Configuration

The Nginx configuration routes traffic to the internal service names. Ensure CORS headers are set for the Explorer UI.

user nginx;
worker_processes auto;

events {
    worker_connections 1024;
}

http {
    upstream explorer_backend {
        server explorer:6060;
    }

    server {
        listen 6060;
        
        location / {
            add_header Access-Control-Allow-Origin * always;
            proxy_pass http://explorer_backend;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

3.2 TDengine Configuration (taos.cfg)

A unified configuration file is mounted into all nodes. Ensure fqdn and firstEp are correctly set.

# First Endpoint
firstEp dnode-01:6030

# Local FQDN configuration (must match hostname for container ID resolution if needed, or specific node FQDN)
# In this dynamic setup, we rely on environment variables or specific host overrides if necessary.
# However, for static mapping in the file:

# Monitor settings
monitor 1
monitorFQDN keeper

# Force config read
forceReadConfig 1

3.3 Adapter and Keeper Settings

In taosadapter.toml, ensure the [uploadKeeper] section points to the keeper service:

[uploadKeeper]
enable = true
url = "http://keeper:6043/adapter_report"

In taoskeeper.toml, configure the connection details:

[tdengine]
host = "adapter"
port = 6041
username = "root"
password = "taosdata"

4. Execution and Deployment

4.1 Launching the Stack

Execute the following commands on the Manager node in sequence:

# Phase 1: Bootstrap
docker stack deploy -c stack-phase-1.yml tdengine-cluster

# Phase 2: Expand Cluster
docker stack deploy -c stack-phase-2.yml tdengine-cluster

# Phase 3: Support Services
docker stack deploy -c stack-phase-3.yml tdengine-cluster

# Phase 4: Gateway
docker stack deploy -c stack-phase-4.yml tdengine-cluster

4.2 Cluster Initialization

Once the containers are running, access the primary dnode to form the TDengine cluster.

# Access the shell of dnode-01
docker exec -it $(docker ps -q -f name=dnode-01) taos

# Inside the TDengine CLI
CREATE DNODE "dnode-02:6030";
CREATE DNODE "dnode-03:6030";

# Verify data nodes
SHOW DNODES;

# Create management nodes for high availability
CREATE MNODE ON DNODE 2;
CREATE MNODE ON DNODE 3;

# Verify management nodes
SHOW MNODES;

5. Monitoring Integration

To enable observability, configure Prometheus to scrape metrics from taosKeeper (port 6043). The taosAdapter pushes internal metrics to taosKeeper as per the configuration in [uploadKeeper].

Access the web interface via the Nginx gateway at http://<ANY_NODE_IP>:6060.