Introduction to Policy Routing

In environments with multiple network interfaces (NICs) and public IP addresses, standard routing often fails to handle outgoing traffic correctly. By default, the Linux kernel routes traffic based solely on the destination address, which can cause packets sent from a secondary interface to be routed through the default gateway of the primary interface. Policy-based routing (PBR) solves this by allowing routing decisions to be based on source addresses or other criteria.

1. Defining Custom Routing Tables

The first step is to assign identifiers to custom routing tables. These entries must be added to the /etc/iproute2/rt_tables file.

Install the necessary tools if they are missing:

apt update && apt install iproute2

Edit the routing tables configuration file:

# /etc/iproute2/rt_tables
#
# reserved values
255     local
254     main
253     default
0       unspec
# local tables
#
100     wan1_table
101     wan2_table

2. Persistent Configuration with Netplan

For modern Ubuntu systems using Netplan, policy routing can be configured directly in the YAML configuration files. This ensures settings persist after a reboot.

Create or edit the configuration file, for example /etc/netplan/01-netcfg.yaml:

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      addresses:
        - 192.168.1.10/24
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8, 1.1.1.1]
      routes:
        - to: 0.0.0.0/0
          via: 192.168.1.1
          table: 100
      routing-policy:
        - from: 192.168.1.10
          table: 100
          priority: 100
    eth1:
      addresses:
        - 10.0.0.10/24
      routes:
        - to: 0.0.0.0/0
          via: 10.0.0.1
          table: 101
      routing-policy:
        - from: 10.0.0.10
          table: 101
          priority: 200

Apply the changes:

netplan apply

In this configuration, each interface has a dedicated routing table. The routing-policy directive ensures that traffic originating from a specific IP looks up the corresponding table.

3. Runtime Configuration Using iproute2

For immediate, non-persistent changes, or for scripts, use the ip command suite.

Setup for the Primary Interface (eth0)

Clear existing entries in the custom table, add the default gateway, ensure local subnet reachability, and add the policy rule.

# Flush the custom table
ip route flush table 100

# Add default route to the custom table
ip route add default via 192.168.1.1 dev eth0 table 100

# Add local subnet route to the custom table (crucial for local switching)
ip route add 192.168.1.0/24 dev eth0 table 100

# Add the policy rule
ip rule add from 192.168.1.10 table 100

Setup for the Secondary Interface (eth1)

ip route flush table 101
ip route add default via 10.0.0.1 dev eth1 table 101
ip route add 10.0.0.0/24 dev eth1 table 101
ip rule add from 10.0.0.10 table 101

Verification

To verify the configuration, inspect the policy rules and the specific routing tables:

# List policy rules
ip rule list

# Show routes for a specific table
ip route show table 100
ip route show table 101

4. Connectivity Testing

To confirm that traffic leaves the correct interface, use curl with the --interface flag or ping with the -I flag.

# Test using source IP of eth0
curl --interface 192.168.1.10 https://ifconfig.me
ping -I 192.168.1.10 8.8.8.8

# Test using source IP of eth1
curl --interface 10.0.0.10 https://ifconfig.me
ping -I 10.0.0.10 8.8.8.8

5. Windows Configuration

On Windows systems, policy routing is typically handled by setting interface metrics or adding specific routes. To prioritize a specific intreface, you can add a persistent route with a lower metric value (lower metric = higher priority).

# Add a persistent route for the secondary network
route -p add 0.0.0.0 mask 0.0.0.0 10.0.0.1 metric 10

# Test connectivity specifying the source address
ping -S 10.0.0.10 8.8.8.8

6. Handling DHCP Interfaces

When using DHCP, you can control the routing priority using metrics or override the routing behavior. Below is a Netplan example where a secondary DHCP interface is configured with a higher metric (lower priority) and forced into a specific routing table.

network:
  version: 2
  ethernets:
    ens5:
      dhcp4: true
      dhcp4-overrides:
        route-metric: 100
    ens6:
      dhcp4: true
      dhcp4-overrides:
        route-metric: 200
      routes:
        - to: 0.0.0.0/0
          via: 10.10.10.1
          table: 102
        - to: 10.10.10.0/24
          table: 102
      routing-policy:
        - from: 10.10.10.50
          table: 102

7. Service Management

In some cases, NetworkManager might conflict with manual network configurations or systemd-networkd. It can be disabled in favor of systemd-networkd.

# Stop and mask NetworkManager
systemctl stop NetworkManager
systemctl disable NetworkManager
systemctl mask NetworkManager

# Enable and start systemd-networkd
systemctl unmask systemd-networkd.service
systemctl enable systemd-networkd.service
systemctl start systemd-networkd.service