Diagnosing and Resolving UDP Packet Loss with iperf3

When experiencing network packet loss, the following optimization strategies are typically employed:
1. NUMA node configuration<br>2. Network interface ring buffer adjustments<br>3. Network interface multi-queue setup<br>4. Network interrupt affinity configuration<br>5. Flow control settings<br>6. Hardware interrupt coalescing<br>7. Kernel TCP/IP parameter tuning<br>8. CPU performance adjustments<br>9. Network driver updates<br>10. Application-level optimizations
1: Testing UDP packet loss with iperf3
1 iperf3 -c target_ip -i 1 -t 300 -u -b 300m<br><br>1. This command initiates a UDP client test with:<br>   - 1-second interval between reports<br>   - UDP protocol testing<br>   - 300-second duration<br>   - 300 Mbps bandwidth limit
1 iperf3 -s<br><br>1. This command starts the server in listening mode
For comprehensive iperf3 command references, consult the official documentation
Through multiple cross-tests, including direct machine connections and connections via switches/routers, approximately 20% UDP packet loss was observed.
1.1 Checking for packet loss on the client side
$cat /proc/net/snmp|grep Udp<br><br>Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti<br>Udp: 911 0 0 601 0 0 0 6<br>UdpLite: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti<br>UdpLite: 0 0 0 0 0 0 0 0
Examine the third (InErrors) and fifth (RcvbufErrors) values in the second Udp line. If both show significant values like 381445, this indicates receiver-side errors, specifically buffer-related issues. Consider increasing rmem_default/rmem_max values. If these values differ, investigate other potential causes by checking ifconfig and netstat -s output.
1.2 Checking for packet loss on the server side
$cat /proc/net/snmp|grep Udp<br>Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti<br>Udp: 42336 1 232345 246 383415 0 0 716<br>UdpLite: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors IgnoredMulti<br>UdpLite: 0 0 0 0 0 0 0 0
Examine the third (InErrors) and fifth (RcvbufErrors) values in the second Udp line. If both show significant values like 381445, this indicates receiver-side errors, specifically buffer-related issues. Consider increasing rmem_default/rmem_max values. If these values differ, investigate other potential causes by checking ifconfig and netstat -s output.
1.3 After identifying whether the issue is client or server-side related:
Use ifconfig to determine which type of packet loss is occurring (errors/dropped/overrun):<br>$ifconfig interface_name<br>eth0: flags=4099 mtu 1500<br>    ether e8:6a:64:48:af:2e txqueuelen 1000 (Ethernet)<br>    RX packets 13103668 bytes 12865579207 (12.8 GB)<br>    RX errors 0 dropped 109 overruns 0 frame 0 (Server-side view)<br>    TX packets 6462181 bytes 5217815587 (5.2 GB)<br>    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 (Client-side view)<br>    device interrupt 16 memory 0xdc200000-dc220000
If errors are non-zero and significant, use ethtool -S interface_name|grep error to identify specific error types for targeted fixes.
If dropped packets are non-zero and significant, investigate protocol layers with netstat -sw, focusing on Ip and Udp fields.
If overrun is non-zero and significant, the network interface's tx/rx queue is too small and requires enlargement.
1.4 For bonded interfaces, consider increasing the txqueuelen of both bonded interfaces to 8192:
$ ifconfig interface_name txqueuelen 8192<br>Verify changes using ip a|grep qlen
1.5 Examine CPU load during iperf3 testing:
Monitor for abnormally high CPU load on specific cores. Determine which metric (%usr, %nice, %sys, %iowait, %irq, %soft, %steal, %guest, %gnice) is elevated to decide if process or interrupt binding is necessary.
$ mpstat -P ALL
Linux 5.4.0-124-generic (hostname) _x86_64 (6 CPU)
21:50:36 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
21:50:36 all 0.06 0.00 0.10 0.24 0.00 0.02 0.00 0.00 0.00 99.58
21:50:36 0 0.05 0.01 0.09 0.24 0.00 0.04 0.00 0.00 0.00 99.58
21:50:36 1 0.05 0.00 0.10 0.17 0.00 0.01 0.00 0.00 0.00 99.67
21:50:36 2 0.06 0.00 0.12 0.23 0.00 0.02 0.00 0.00 0.00 99.58
21:50:36 3 0.06 0.00 0.11 0.41 0.00 0.01 0.00 0.00 0.00 99.41
21:50:36 4 0.07 0.01 0.11 0.20 0.00 0.01 0.00 0.00 0.00 99.61
21:50:36 5 0.06 0.01 0.08 0.17 0.00 0.05 0.00 0.00 0.00 99.64
2 Solutions
Through extensive testing, the solution was found to be modifying kernel parameters. After these changes, UDP packet loss was eliminated:
Before modification:<br>sysctl net.core.rmem_default=213442<br>sysctl net.core.rmem_max=2062152<br>sysctl net.ipv4.udp_mem="756207  1024438  1536424"
After modification:<br>sysctl net.core.rmem_default=21344200<br>sysctl net.core.rmem_max=206215200<br>sysctl net.ipv4.udp_mem="75620700  102443800  153642400"
For detailed kernel parameter information, refer to official kernel documentation
Note: Optimized parameters will consume significant memory. If memory is insufficient, consider increasing physical RAM capacity.

Tags: network Performance udp packet loss iperf3

Posted on Mon, 18 May 2026 22:53:26 +0000 by tkm

Hot Tags

©2026 Freaks City