Overview
This guide demonstrates three methods for connecting container networks across different nodes using Linux iproute2 tools. These approaches replicate the backend capabilities commonly used in container orchestration platforms.
The three methods covered include:
- host-gw (host gateway mode)
- ipip (IP-in-IP tunneling)
- vxlan (Virtual Extensible LAN)
Test Environment Setup
The test environment uses network namespaces to isolate network resources, simulating container network stacks. Two virtual machines serve as the physical hosts:
- node-a (192.168.32.100) - First node
- node-b (192.168.32.101) - Second node
Network objective: Connect the 172.100.0.0/24 subnet on node-a with the 172.101.0.0/24 subnet on node-b.
┌────────────────────────────────┐ ┌────────────────────────────────┐
│ node-a │ │ node-b │
│ ┌──────────────────┐ │ │ ┌──────────────────┐ │
│ │ ns-net100 │ │ │ │ ns-net101 │ │
│ │ 172.100.0.2 │ │ │ │ 172.101.0.2 │ │
│ │ eth0 │ │ │ │ eth0 │ │
│ └────────┬────────┘ │ │ └────────┬────────┘ │
│ │ │ │ │ │
│ veth100 │ │ veth101 │
│ │ │ │ │ │
│ br100 │ │ br101 │
│ 172.100.0.1/24 │ │ 172.101.0.1/24 │
│ │ │ │
│ 192.168.32.100/24 │ │ 192.168.32.101/24 │
│ eth0 │ │ eth0 │
└──────────────┬───────────────┘ └──────────────┬──────────────┘
│ │
└────────────────────────────────────────┘
Prerequisites
Enable IP forwarding to allow traffic to pass through bridges into namespaces:
sysctl -w net.ipv4.ip_forward=1
Configuring node-a (192.168.32.100)
Create and configure the bridge interface:
# Create bridge
ip link add br100 type bridge
# Activate bridge
ip link set br100 up
Create network namespace and veth pair:
# Create network namespace
ip netns add ns-net100
# Create veth pair, moving one end into namespace
ip link add veth100 type veth peer name eth0 netns ns-net100
# Enable interface inside namespace
ip netns exec ns-net100 ip link set eth0 up
# Enable veth on host
ip link set veth100 up
# Attach veth to bridge
ip link set veth100 master br100
Configure IP addresses and routing:
# Assign bridge IP address
ip addr add 172.100.0.1/24 dev br100
# Assign container IP address
ip netns exec ns-net100 ip addr add 172.100.0.2/24 dev eth0
# Set default route via bridge
ip netns exec ns-net100 ip route add default via 172.100.0.1 dev eth0
For node-b (192.168.32.101), repeat these steps with 100 and 101 values swapped appropriately.
After configuration, node-a network interface summary:
$ ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:ce:51:a5 brd ff:ff:ff:ff:ff:ff
inet 192.168.32.100/24 brd 192.168.32.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
9: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 02:8d:b1:91:fa:7a brd ff:ff:ff:ff:ff:ff
inet 172.100.0.1/24 scope global br100
valid_lft forever preferred_lftforever
$ ip netns exec ns-net100 ip addr
2: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether be:9c:d4:57:58:5a brd ff:ff:ff:ff:ff:ff
inet 172.100.0.2/24 scope global eth0
valid_lft forever preferred_lft forever
Method 1: host-gw (Host Gateway)
The host-gw approach requires only a single route pointing to the remote node.
# On node-a
ip route add 172.101.0.0/24 via 192.168.32.101 dev eth0 onlink
# On node-b
ip route add 172.100.0.0/24 via 192.168.32.100 dev eth0 onlink
Test connectivity from the namespace:
$ ip netns exec ns-net100 ping -c 1 172.101.0.2
PING 172.101.0.2 (172.101.0.2) 56(84) bytes of data.
64 bytes from 172.101.0.2: icmp_seq=1 ttl=62 time=0.656 ms
1 packets transmitted, 1 received, 0% packet loss, time 0ms
Traffic flow for host-gw mode:
172.100.0.2 → ns-net100:eth0 → br100 → node-a:eth0 → node-b:eth0 → br101 → ns-net101:eth0 → 172.101.0.2
Q&A
Q: Why must containers using host-gw mode be on the same L2 network?
A: The routing does not modify the source IP. When returning packets swap source and destination, the gateway queries the routing table but cannot determine the outgoing interface. On the same L2 network, this is not an issue because MAC addresses are used for forwarding instead of routing tables.
Method 2: ipip (IP-in-IP Tunneling)
IPIP is a network layer tunneling protocol that encapsulates IP packets within another IP packet. This enables transparent communication between networks without modifying the inner packet contents.
Packet format comparison:
Original packet:
+--------------------------------+
| Application |
+--------------------------------+
| TCP/UDP |
+--------------------------------+
| IP Header |
+--------------------------------+
| Ethernet |
+--------------------------------+
After IPIP encapsulation:
+--------------------------------+
| Application |
+--------------------------------+
| TCP/UDP |
+--------------------------------+
| Inner IP Header |
+--------------------------------+
| Outer IP Header (Tunnel) |
+--------------------------------+
| Ethernet |
+--------------------------------+
Configuration on node-a
# Create IPIP tunnel interface
ip tunnel add tun100 mode ipip local 192.168.32.100 remote 192.168.32.101
# Assign tunnel IP address
ip addr add 172.100.1.1/30 dev tun100
# Enable tunnel interface
ip link set tun100 up
# Route remote subnet through tunnel
ip route add 172.101.0.0/24 via 172.100.1.2 dev tun100
For node-b, swap 100 and 101 in the IP addresses.
Test connectivity:
$ ip netns exec ns-net101 ping -c 1 172.100.0.2
Capture packets on the physical interface:
$ tcpdump -s0 -i eth0 host 192.168.32.101 -nn
11:28:01.268176 IP 192.168.32.101 > 192.168.32.100: ICMP echo request
11:28:01.268299 IP 192.168.32.100 > 192.168.32.101: ICMP echo reply
11:28:08.697558 IP 192.168.32.101 > 192.168.32.100: IP 172.101.0.2 > 172.100.0.2: ICMP echo request (ipip-proto-4)
11:28:08.697760 IP 192.168.32.100 > 192.168.32.101: IP 172.100.0.2 > 172.101.0.2: ICMP echo reply (ipip-proto-4)
Decaptured view shows the IPIP header wrapping the inner packet:
Frame: 118 bytes captured
Ethernet II, Src: 52:54:00:b9:5d:53, Dst: 52:54:00:ce:51:a5
Internet Protocol Version 4, Src: 192.168.32.101, Dst: 192.168.32.100
Internet Protocol Version 4, Src: 172.101.0.2, Dst: 172.100.0.2
Internet Control Message Protocol
Traffic flow with IPIP encapsulation:
br100 → tun100 → node-a:eth0 → node-b:eth0 → tun101 → br101 → 172.101.0.2
| | | | | |
| | encaps | decaps| | |
Q&A
Q: What is the main difference from host-gw?
A: IPIP removes the requirement for L2 connectivity between nodes. Only L3 connectivity is needed between the physical networks.
Method 3: vxlan (Virtual Extensible LAN)
VXLAN addresses VLAN limitations in large cloud data centers and multi-tenant environments. By encapsulating L2 Ethernet frames in UDP, VXLAN enables L2 overlay networks across L3 infrastructure, supporting up to 16 million virtual networks compared to VLAN's 4096 limit.
VXLAN packet structure:
Original packet:
+--------------------------------+
| Application |
+--------------------------------+
| TCP/UDP |
+--------------------------------+
| IP Header |
+--------------------------------+
| Ethernet |
+--------------------------------+
After VXLAN encapsulation:
+--------------------------------+
| Application |
+--------------------------------+
| TCP/UDP |
+--------------------------------+
| IP Header |
+--------------------------------+
| Ethernet |
+--------------------------------+
/ VXLAN Header (8 bytes) /
+--------------------------------+
/ UDP Header /
+--------------------------------+
/ Outer IP Header (Tunnel) /
+--------------------------------+
/ Outer Ethernet (Tunnel) /
+--------------------------------+
Configuration on node-a
# Create VXLAN tunnel endpoint
ip link add vtep100 type vxlan id 1 local 192.168.32.100 dev eth0 dstport 8472 nolearning
# Assign VXLAN endpoint address
ip addr add 172.100.0.0/32 dev vtep100
# Set appropriate MTU
ip link set vtep100 mtu 1450
# Enable interface
ip link set vtep100 up
# Configure routing via VXLAN
ip route add 172.101.0.0/24 via 172.101.0.0 dev vtep100 onlink
For node-b, swap 100 and 101 in the configuraton.
Verify VXLAN interface details:
$ ip -d link show vtep100
12: vtep100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether 52:3a:8c:c9:08:e8 brd ff:ff:ff:ff:ff:ff
vxlan id 1 local 192.168.32.100 dev eth0 srcport 0 0 dstport 8472 nolearning
Verify UDP listener:
$ netstat -nulp
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 0 0 0.0.0.0:8472 0.0.0.0:*
Attempt initial connectivity test:
$ ip netns exec ns-net100 ping -c 1 172.101.0.2
Monitor traffic on VXLAN interface:
$ tcpdump -s0 -i vtep100 -nn
17:44:26.916277 ARP, Request who-has 172.101.0.0 tell 172.100.0.0, length 28
The ARP request indicates that the gateway MAC adress is unknown. Configure static neighbor entries:
# On node-a
ip neigh add 172.101.0.0 dev vtep100 lladdr 02:38:4a:12:3b:43
# On node-b
ip neigh add 172.100.0.0 dev vtep101 lladdr 52:3a:8c:c9:08:e8
Configure forwarding database entries to direct VXLAN traffic to the correct node:
# On node-a
bridge fdb add 02:38:4a:12:3b:43 dev vtep100 dst 192.168.32.101
# On node-b
bridge fdb add 52:3a:8c:c9:08:e8 dev vtep101 dst 192.168.32.100
Test connectivity again:
$ ip netns exec ns-net100 ping -c 1 172.101.0.2
64 bytes from 172.101.0.2: icmp_seq=1 ttl=62 time=1.2 ms
VXLAN traffic flow with encapsulation:
br100 → vtep100 → node-a:eth0 (UDP 8472) → node-b:eth0 → vtep101 → br101 → 172.101.0.2
| | | | |
| | encaps | decaps |
Q&A
Q: Why use 172.101.0.0 as the gateway address on node-a?
A: The gateway must not be an IP address assigned to the local node or within its subnet. Using a non-routable network identifier prevents address conflicts and ensures proper forwarding to the remote node.
Comparison Summary
Functional Differences
| Mode | Complexity | Connectivity Requirement |
|---|---|---|
| host-gw | Minimal (single route) | L2 (same broadcast domain) |
| ipip | Medium (tunnel interface) | L3 (IP reachability) |
| vxlan | High (overlay network) | L3 (IP reachability) |
Performance Characteristics
- host-gw: Fastest - no encapsulation overhead
- ipip: Moderate - adds 20 bytes per packet (outer IP header)
- vxlan: Slowest - adds Ethernet + IP + UDP + VXLAN headers (~50+ bytes)
Use Case Recommendations
- Use host-gw when nodes are on the same L2 network and maximum performance is needed
- Use ipip when nodes span different L2 networks but remain in the same datacenter
- Use vxlan for multi-tenant environments requiring network isolation or across geographically distributed nodes
References
- RFC 7348 - Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks