VyOS 1.5, Segment Routing & GRE
I was reading about Segment Routing, and I wanted to give that technology a try.
I already had a GNS3 lab built with vSRX devices, with a basic IS-IS+LDP topology, running iBGP between PE routers and a L3VPN service. Unfortunately after trying to enable SR on the vSRX devices I found that you cannot configure SRGB on them. I'd have to re-do my lab with vMX and that's not fun for me, it takes it out of "weekend fun project" territory and more into "homework".
So I thought, why not deploy Segment Routing on my own network?. Not directly on AS203528 yet (I don't want to mess with it due to a current outage I have), but on my internal network. It all runs VyOS, IGP IS-IS and iBGP (with 4 RRs). There's no MPLS deployed there yet. After all this network is supposed to be a Lab (even though it has taken more of a "production" role).
Initial deployment
First of all I updated all the nodes (except RRs) on my Internal VyOS network to the latest 1.5 nightly at the time (1.5-rolling-202312290919). Easy enough. All good so far.
Then it was just enabling SR within IS-IS (It already has default SRGB/SRLB values) and assigning on each router the index values for the loopback prefixes. And also enabling MPLS on the internally-facing interfaces.
Here is an example, first are the loopback IPs of one of my routers, and then the deployed config to enable SR.
1dum0 192.168.254.34/32 xx:xx:xx:xx:xx:xx default 1500 u/u Loopback / Tunnel source 2 2a0e:8f02:21d2:ffff::34/128 3dum1 fc0e:8f02:21d2:ffff::34/128 xx:xx:xx:xx:xx:xx default 1500 u/u IPv6 iBGP next hop 4 5 6set protocols isis segment-routing maximum-label-depth '15' 7set protocols isis segment-routing prefix 2a0e:8f02:21d1:ffff::34/128 index value '341' 8set protocols isis segment-routing prefix 192.168.254.34/32 index value '340' 9set protocols isis segment-routing prefix fc0e:8f02:21d1:ffff::34/128 index value '342' 10 11set protocols mpls interface 'eth0' 12set protocols mpls interface 'eth1'
After deploying that everywhere - it just worked:
1OSR2A1:~$ sh ip route 192.168.254.13 2Routing entry for 192.168.254.13/32 3 Known via "isis", distance 115, metric 45210, best 4 Last update 1d00h32m ago 5 * 172.27.19.17, via eth1, label 16130, weight 1 6 7OSR2A1:~$ sh ipv6 route 2a0e:8f02:21d1:ffff::13 8Routing entry for 2a0e:8f02:21d1:ffff::13/128 9 Known via "isis", distance 115, metric 45210, best 10 Last update 1d00h33m ago 11 * fe80::9434:bdff:fe26:3f79, via eth1, label 16131, weight 1 12 13OSR2A1:~$ sh ipv6 route fc0e:8f02:21d1:ffff::13 14Routing entry for fc0e:8f02:21d1:ffff::13/128 15 Known via "isis", distance 115, metric 45210, best 16 Last update 1d00h33m ago 17 * fe80::9434:bdff:fe26:3f79, via eth1, label 16132, weight 1
Problems
It all worked nice in the beginning, or so I thought. Later on the day I did find that there was some really sporadic performance degradation happening on my network, to/from the IPv6 Internet.
My internal network is connected to the Public v6 Internet through a redundant set of firewalls (OSR1FW1/OSR1FW2), both of these are on different servers at home.
Each of these firewalls is connected to two Border routers I have at home, OSR1BR1/OSR1BR2. The connection between Firewall & Border Router on the same server is through a local VLAN. However the connection to the FW/BR on the different server is through a GRETAP tunnel traversing my core.
The active firewall will do ECMP to/from both of the border routers. So that would partly explain the inconsistent experience and degradation.
Below is the example configuration of such a GRETAP tunnel across my core, for the Firewall to BR connection: (OSR1CR4 side looks pretty much the same)
1OSR1CR2# sh interfaces ethernet eth12 2 description "OSR1FW2 - OSR1BR1 VLAN 543" 3 offload { 4 gro 5 gso 6 sg 7 tso 8 } 9 10 11OSR1CR2# sh interfaces bridge br2 12 description "OSR1FW2 - OSR1BR1 VLAN 543" 13 enable-vlan 14 ipv6 { 15 address { 16 no-default-link-local 17 } 18 } 19 member { 20 interface eth12 { 21 allowed-vlan 100 22 native-vlan 100 23 } 24 interface tun2 { 25 allowed-vlan 100 26 native-vlan 100 27 } 28 } 29 30OSR1CR2# sh interfaces tunnel tun2 31 description "OSR1FW2 - OSR1BR1 VLAN 543" 32 encapsulation gretap 33 mtu 1600 34 parameters { 35 ip { 36 key 543 37 } 38 } 39 remote 192.168.254.13 40 source-address 192.168.254.11
I noticed that, on the core router where the tunnel exists, whenever the performance issue was seen, the core-facing interface (in this case OSR1CR2 to OSR1CR1) would drop packets on TX.:
1fabrizzio@OSR1CR2:~$ sh interfaces ethernet eth0 2eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1800 qdisc mq state UP group default qlen 1000 3 link/ether f6:65:87:96:70:22 brd ff:ff:ff:ff:ff:ff 4 altname enp0s18 5 altname ens18 6 inet 172.27.16.38/30 brd 172.27.16.39 scope global eth0 7 valid_lft forever preferred_lft forever 8 inet6 2a0e:8f02:21d1:feed:0:1:10:12/126 scope global 9 valid_lft forever preferred_lft forever 10 inet6 fe80::f465:87ff:fe96:7022/64 scope link 11 valid_lft forever preferred_lft forever 12 Description: To OSR1CR1 13 14 RX: bytes packets errors dropped overrun mcast 15 1261617086 3428022 0 20 0 0 16 TX: bytes packets errors dropped carrier collisions 17 5598511155 6117186 0 5212 0 0 <<<<<<<<<<<<<<< 18 19fabrizzio@OSR1CR2:~$ sh ip route 192.168.254.13 20Routing entry for 192.168.254.13/32 21 Known via "isis", distance 115, metric 1210, best 22 Last update 14:42:50 ago 23 * 172.27.16.37, via eth0, label 16130, weight 1
Here is an IPerf3 test running OSR1FW2 <> OSR1BR1, to test the performance across the GRETAP tunnel. The results are pretty bad, this should be close to 2 Gbit/s. Tons of retransmissions and SACK's were seen. The tcpdump was taken at OSR1CR4 interface (facing OSR1FW2) and it's visible that the traffic enters OSR1CR4 however the far end is missing some segments.
1OSR1FW2:~$ iperf3 -c 172.27.1.17 2Connecting to host 172.27.1.17, port 5201 3[ 5] local 172.27.1.18 port 39192 connected to 172.27.1.17 port 5201 4[ ID] Interval Transfer Bitrate Retr Cwnd 5[ 5] 0.00-1.00 sec 139 KBytes 1.13 Mbits/sec 43 8.48 KBytes 6[ 5] 1.00-2.00 sec 45.2 KBytes 371 Kbits/sec 40 8.48 KBytes 7[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 38 8.48 KBytes 8^C- - - - - - - - - - - - - - - - - - - - - - - - - 9[ ID] Interval Transfer Bitrate Retr 10[ 5] 0.00-3.09 sec 184 KBytes 487 Kbits/sec 121 sender 11[ 5] 0.00-3.09 sec 0.00 Bytes 0.00 bits/sec receiver 12iperf3: interrupt - the client has terminated 13 14 1516:39:41.862143 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [S], seq 1766734757, win 64240, options [mss 1460,sackOK,TS val 2424729071 ecr 0,nop,wscale 7], length 0 1616:39:41.862850 IP 172.27.1.17.5201 > 172.27.1.18.39192: Flags [S.], seq 3455293594, ack 1766734758, win 65160, options [mss 1460,sackOK,TS val 4290898765 ecr 2424729071,nop,wscale 7], length 0 1716:39:41.863012 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [.], ack 1, win 502, options [nop,nop,TS val 2424729072 ecr 4290898765], length 0 1816:39:41.863051 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [P.], seq 1:38, ack 1, win 502, options [nop,nop,TS val 2424729072 ecr 4290898765], length 37 1916:39:41.863749 IP 172.27.1.17.5201 > 172.27.1.18.39192: Flags [.], ack 38, win 509, options [nop,nop,TS val 4290898766 ecr 2424729072], length 0 2016:39:41.864481 IP 172.27.1.17.5201 > 172.27.1.18.39180: Flags [P.], seq 3:4, ack 166, win 508, options [nop,nop,TS val 4290898766 ecr 2424729070], length 1 2116:39:41.864492 IP 172.27.1.17.5201 > 172.27.1.18.39180: Flags [P.], seq 4:5, ack 166, win 508, options [nop,nop,TS val 4290898766 ecr 2424729070], length 1 2216:39:41.864616 IP 172.27.1.18.39180 > 172.27.1.17.5201: Flags [.], ack 5, win 502, options [nop,nop,TS val 2424729073 ecr 4290898764], length 0 2316:39:41.864658 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [P.], seq 38:7278, ack 1, win 502, options [nop,nop,TS val 2424729073 ecr 4290898766], length 7240 2416:39:41.864700 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [P.], seq 7278:14518, ack 1, win 502, options [nop,nop,TS val 2424729073 ecr 4290898766], length 7240 2516:39:41.864773 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [.], seq 14518:15966, ack 1, win 502, options [nop,nop,TS val 2424729074 ecr 4290898766], length 1448 2616:39:41.865324 IP 172.27.1.17.5201 > 172.27.1.18.39192: Flags [.], ack 38, win 509, options [nop,nop,TS val 4290898767 ecr 2424729072,nop,nop,sack 1 {14518:15966}], length 0 2716:39:41.865437 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [.], seq 38:1486, ack 1, win 502, options [nop,nop,TS val 2424729074 ecr 4290898767], length 1448 2816:39:41.866101 IP 172.27.1.17.5201 > 172.27.1.18.39192: Flags [.], ack 1486, win 498, options [nop,nop,TS val 4290898768 ecr 2424729074,nop,nop,sack 1 {14518:15966}], length 0 2916:39:41.866222 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [P.], seq 1486:7278, ack 1, win 502, options [nop,nop,TS val 2424729075 ecr 4290898768], length 5792 3016:39:42.070774 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [.], seq 1486:2934, ack 1, win 502, options [nop,nop,TS val 2424729280 ecr 4290898768], length 1448 3116:39:42.071778 IP 172.27.1.17.5201 > 172.27.1.18.39192: Flags [.], ack 2934, win 490, options [nop,nop,TS val 4290898974 ecr 2424729280,nop,nop,sack 1 {14518:15966}], length 0 3216:39:42.072008 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [P.], seq 2934:7278, ack 1, win 502, options [nop,nop,TS val 2424729281 ecr 4290898974], length 4344 3316:39:42.072042 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [.], seq 7278:8726, ack 1, win 502, options [nop,nop,TS val 2424729281 ecr 4290898974], length 1448 3416:39:42.072743 IP 172.27.1.17.5201 > 172.27.1.18.39192: Flags [.], ack 2934, win 490, options [nop,nop,TS val 4290898975 ecr 2424729280,nop,nop,sack 2 {7278:8726}{14518:15966}], length 0 3516:39:42.072898 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [.], seq 2934:4382, ack 1, win 502, options [nop,nop,TS val 2424729282 ecr 4290898975], length 1448 3616:39:42.073582 IP 172.27.1.17.5201 > 172.27.1.18.39192: Flags [.], ack 4382, win 479, options [nop,nop,TS val 4290898975 ecr 2424729282,nop,nop,sack 2 {7278:8726}{14518:15966}], length 0 3716:39:42.073776 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [P.], seq 4382:7278, ack 1, win 502, options [nop,nop,TS val 2424729283 ecr 4290898975], length 2896 3816:39:42.073817 IP 172.27.1.18.39192 > 172.27.1.17.5201: Flags [P.], seq 15966:18862, ack 1, win 502, options [nop,nop,TS val 2424729283 ecr 4290898975], length 2896
The odd thing is if I repeat the capture on the OSR1CR4 interface facing the core (OSR1CR3) we see that the segments from OSR1FW2 to OSR1BR1 are indeed already missing and not being sent out towards the Core.
The weird tcpdump filter is because from OSR1CR4 towards GRE destination OSR1CR2 we use MPLS label 16110. Due to penultimate hop popping (my hands keep typing pooping by themselves :D ), the traffic on the reverse direction will arrive without a label; I need to be able to capture both directions.
1fabrizzio@OSR1CR4:~$ sh ip route 192.168.254.11 2Routing entry for 192.168.254.11/32 3 Known via "isis", distance 115, metric 1210, best 4 Last update 19:23:51 ago 5 * 172.27.16.41, via eth0, label 16110, weight 1 6 7root@OSR1CR4:~# tcpdump -i eth0 "(src 192.168.254.11 && dst 192.168.254.13) or (mpls 16110 && (src 192.168.254.13 && dst 192.168.254.11))" | grep 0x21f 8 9<snipped> 10 1116:45:49.477938 MPLS (label 16110, tc 0, [S], ttl 64) IP 192.168.254.13 > 192.168.254.11: GREv0, key=0x21f, length 82: IP 172.27.1.18.37834 > 172.27.1.17.5201: Flags [S], seq 1179634274, win 64240, options [mss 1460,sackOK,TS val 2425096687 ecr 0,nop,wscale 7], length 0 1216:45:49.478705 IP 192.168.254.11 > 192.168.254.13: GREv0, key=0x21f, length 82: IP 172.27.1.17.5201 > 172.27.1.18.37834: Flags [S.], seq 3242204630, ack 1179634275, win 65160, options [mss 1460,sackOK,TS val 4291266381 ecr 2425096687,nop,wscale 7], length 0 1316:45:49.478851 MPLS (label 16110, tc 0, [S], ttl 64) IP 192.168.254.13 > 192.168.254.11: GREv0, key=0x21f, length 74: IP 172.27.1.18.37834 > 172.27.1.17.5201: Flags [.], ack 1, win 502, options [nop,nop,TS val 2425096688 ecr 4291266381], length 0 1416:45:49.478865 MPLS (label 16110, tc 0, [S], ttl 64) IP 192.168.254.13 > 192.168.254.11: GREv0, key=0x21f, length 111: IP 172.27.1.18.37834 > 172.27.1.17.5201: Flags [P.], seq 1:38, ack 1, win 502, options [nop,nop,TS val 2425096688 ecr 4291266381], length 37 1516:45:49.479638 IP 192.168.254.11 > 192.168.254.13: GREv0, key=0x21f, length 74: IP 172.27.1.17.5201 > 172.27.1.18.37834: Flags [.], ack 38, win 509, options [nop,nop,TS val 4291266382 ecr 2425096688], length 0 1616:45:49.480309 IP 192.168.254.11 > 192.168.254.13: GREv0, key=0x21f, length 75: IP 172.27.1.17.5201 > 172.27.1.18.37828: Flags [P.], seq 3:4, ack 166, win 508, options [nop,nop,TS val 4291266382 ecr 2425096686], length 1 1716:45:49.480330 IP 192.168.254.11 > 192.168.254.13: GREv0, key=0x21f, length 75: IP 172.27.1.17.5201 > 172.27.1.18.37828: Flags [P.], seq 4:5, ack 166, win 508, options [nop,nop,TS val 4291266382 ecr 2425096686], length 1 1816:45:49.480471 MPLS (label 16110, tc 0, [S], ttl 64) IP 192.168.254.13 > 192.168.254.11: GREv0, key=0x21f, length 74: IP 172.27.1.18.37828 > 172.27.1.17.5201: Flags [.], ack 5, win 502, options [nop,nop,TS val 2425096689 ecr 4291266379], length 0 1916:45:49.480677 MPLS (label 16110, tc 0, [S], ttl 64) IP 192.168.254.13 > 192.168.254.11: GREv0, key=0x21f, length 1522: IP 172.27.1.18.37834 > 172.27.1.17.5201: Flags [.], seq 14518:15966, ack 1, win 502, options [nop,nop,TS val 2425096689 ecr 4291266382], length 1448 2016:45:49.481335 IP 192.168.254.11 > 192.168.254.13: GREv0, key=0x21f, length 86: IP 172.27.1.17.5201 > 172.27.1.18.37834: Flags [.], ack 38, win 509, options [nop,nop,TS val 4291266383 ecr 2425096688,nop,nop,sack 1 {14518:15966}], length 0 2116:45:49.481478 MPLS (label 16110, tc 0, [S], ttl 64) IP 192.168.254.13 > 192.168.254.11: GREv0, key=0x21f, length 1522: IP 172.27.1.18.37834 > 172.27.1.17.5201: Flags [.], seq 38:1486, ack 1, win 502, options [nop,nop,TS val 2425096690 ecr 4291266383], length 1448 2216:45:49.482121 IP 192.168.254.11 > 192.168.254.13: GREv0, key=0x21f, length 86: IP 172.27.1.17.5201 > 172.27.1.18.37834: Flags [.], ack 1486, win 498, options [nop,nop,TS val 4291266384 ecr 2425096690,nop,nop,sack 1 {14518:15966}], length 0 2316:45:49.686840 MPLS (label 16110, tc 0, [S], ttl 64) IP 192.168.254.13 > 192.168.254.11: GREv0, key=0x21f, length 1522: IP 172.27.1.18.37834 > 172.27.1.17.5201: Flags [.], seq 1486:2934, ack 1, win 502, options [nop,nop,TS val 2425096896 ecr 4291266384], length 1448 2416:45:49.687786 IP 192.168.254.11 > 192.168.254.13: GREv0, key=0x21f, length 86: IP 172.27.1.17.5201 > 172.27.1.18.37834: Flags [.], ack 2934, win 490, options [nop,nop,TS val 4291266590 ecr 2425096896,nop,nop,sack 1 {14518:15966}], length 0 2516:45:49.687950 MPLS (label 16110, tc 0, [S], ttl 64) IP 192.168.254.13 > 192.168.254.11: GREv0, key=0x21f, length 1522: IP 172.27.1.18.37834 > 172.27.1.17.5201: Flags [.], seq 2934:4382, ack 1, win 502, options [nop,nop,TS val 2425096897 ecr 4291266590], length 1448 2616:45:49.688011 MPLS (label 16110, tc 0, [S], ttl 64) IP 192.168.254.13 > 192.168.254.11: GREv0, key=0x21f, length 1522: IP 172.27.1.18.37834 > 172.27.1.17.5201: Flags [.], seq 4382:5830, ack 1, win 502, options [nop,nop,TS val 2425096897 ecr 4291266590], length 1448
Troubleshooting
First of all I disabled all enabled offloads via VyOS on the core-facing interface:
1fabrizzio@OSR1CR4:~$ configure 2[edit] 3fabrizzio@OSR1CR4# delete interfaces ethernet eth0 offload 4[edit] 5fabrizzio@OSR1CR4# commit
This did not fix the issue.
1OSR1CR4:~$ sh interfaces ethernet eth0 2eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1800 qdisc mq state UP group default qlen 1000 3 link/ether 3a:e6:63:fc:e9:68 brd ff:ff:ff:ff:ff:ff 4 altname enp0s18 5 altname ens18 6 inet 172.27.16.42/30 brd 172.27.16.43 scope global eth0 7 valid_lft forever preferred_lft forever 8 inet6 2a0e:8f02:21d1:feed:0:1:11:12/126 scope global 9 valid_lft forever preferred_lft forever 10 inet6 fe80::38e6:63ff:fefc:e968/64 scope link 11 valid_lft forever preferred_lft forever 12 Description: To OSR1CR3 13 14 RX: bytes packets errors dropped overrun mcast 15 7400341272 9874881 0 28 0 0 16 TX: bytes packets errors dropped carrier collisions 17 1002401375 4853418 0 36577 0 0 <<<<
Ethtool shows no queue drops
1fabrizzio@OSR1CR4:~$ ethtool -S eth0 2NIC statistics: 3 rx_queue_0_packets: 9878837 4 rx_queue_0_bytes: 7405521629 5 rx_queue_0_drops: 0 6 rx_queue_0_xdp_packets: 0 7 rx_queue_0_xdp_tx: 0 8 rx_queue_0_xdp_redirects: 0 9 rx_queue_0_xdp_drops: 0 10 rx_queue_0_kicks: 2066 11 tx_queue_0_packets: 4853851 12 tx_queue_0_bytes: 1002468305 13 tx_queue_0_xdp_tx: 0 14 tx_queue_0_xdp_tx_drops: 0 15 tx_queue_0_kicks: 4595142 16 tx_queue_0_tx_timeouts: 0 17 18fabrizzio@OSR1CR4:~$ ethtool -k eth0 19Features for eth0: 20rx-checksumming: on [fixed] 21tx-checksumming: on 22 tx-checksum-ipv4: off [fixed] 23 tx-checksum-ip-generic: on 24 tx-checksum-ipv6: off [fixed] 25 tx-checksum-fcoe-crc: off [fixed] 26 tx-checksum-sctp: off [fixed] 27scatter-gather: off 28 tx-scatter-gather: off 29 tx-scatter-gather-fraglist: off [fixed] 30tcp-segmentation-offload: off 31 tx-tcp-segmentation: off 32 tx-tcp-ecn-segmentation: off 33 tx-tcp-mangleid-segmentation: off 34 tx-tcp6-segmentation: off 35generic-segmentation-offload: off 36generic-receive-offload: off 37large-receive-offload: off [fixed] 38rx-vlan-offload: off [fixed] 39tx-vlan-offload: off [fixed] 40ntuple-filters: off [fixed] 41receive-hashing: off [fixed] 42highdma: on [fixed] 43rx-vlan-filter: on [fixed] 44vlan-challenged: off [fixed] 45tx-lockless: off [fixed] 46netns-local: off [fixed] 47tx-gso-robust: on [fixed] 48tx-fcoe-segmentation: off [fixed] 49tx-gre-segmentation: off [fixed] 50tx-gre-csum-segmentation: off [fixed] 51tx-ipxip4-segmentation: off [fixed] 52tx-ipxip6-segmentation: off [fixed] 53tx-udp_tnl-segmentation: off [fixed] 54tx-udp_tnl-csum-segmentation: off [fixed] 55tx-gso-partial: off [fixed] 56tx-tunnel-remcsum-segmentation: off [fixed] 57tx-sctp-segmentation: off [fixed] 58tx-esp-segmentation: off [fixed] 59tx-udp-segmentation: off [fixed] 60tx-gso-list: off [fixed] 61fcoe-mtu: off [fixed] 62tx-nocache-copy: off 63loopback: off [fixed] 64rx-fcs: off [fixed] 65rx-all: off [fixed] 66tx-vlan-stag-hw-insert: off [fixed] 67rx-vlan-stag-hw-parse: off [fixed] 68rx-vlan-stag-filter: off [fixed] 69l2-fwd-offload: off [fixed] 70hw-tc-offload: off [fixed] 71esp-hw-offload: off [fixed] 72esp-tx-csum-hw-offload: off [fixed] 73rx-udp_tunnel-port-offload: off [fixed] 74tls-hw-tx-offload: off [fixed] 75tls-hw-rx-offload: off [fixed] 76rx-gro-hw: on 77tls-hw-record: off [fixed] 78rx-gro-list: off 79macsec-hw-offload: off [fixed] 80rx-udp-gro-forwarding: off 81hsr-tag-ins-offload: off [fixed] 82hsr-tag-rm-offload: off [fixed] 83hsr-fwd-offload: off [fixed] 84hsr-dup-offload: off [fixed]
Seeing this, I had to dig deeper and do some research. I found something related to MPLS + offloading drops on LKML and openvswitch as well as a tangentially-related blog post at Cloudflare
I decided that the best path forward would be to try and figure out if/why is the kernel dropping the packets. I added the Debian Bookworm repos onto OSR1CR2 (far end of GRE tunnel) and installed dropwatch.
I ran it with dropwatch -l kas, then configured alertmode packet set alertmode packet, saw what was normal (lots of ICMPv6 drops...) and re-ran the IPerf3 test. This is what started popping out. Protocol 0x8847 (MPLS Unicast) is a good hint that it's the traffic I care about. Length is big, maybe due to the various offloads doing their things.
1drop at: validate_xmit_skb+0x29c/0x320 (0xffffffff8d6b1a6c) 2origin: software 3timestamp: Mon Jan 1 11:52:45 2024 028417108 nsec 4protocol: 0x8847 5length: 3008 6original length: 3008 7drop reason: NOT_SPECIFIED
After digging around using Google, I found this which pointed me to offloads being the potential culprit. I had already disabled offloads on the core-facing interface that is actually dropping the packets.
I tried disabling the offloads on the interface at OSR1CR4 facing OSR1FW2. Also no change on the performance.
The last option I had was to check both the Bridge tying up everything and the tunnel interface. They have no offloads configured.
1fabrizzio@OSR1CR4# show interfaces bridge br2 2 description "OSR1FW2 - OSR1BR1 VLAN 543" 3 enable-vlan 4 ipv6 { 5 address { 6 no-default-link-local 7 } 8 } 9 member { 10 interface eth10 { 11 allowed-vlan 100 12 native-vlan 100 13 } 14 interface tun2 { 15 allowed-vlan 100 16 native-vlan 100 17 } 18 } 19[edit]
However the tunnel interface on VyOS does have several offloads enabled by default:
1fabrizzio@OSR1CR4:~$ ethtool -k tun2 2Features for tun2: 3rx-checksumming: off [fixed] 4tx-checksumming: on 5 tx-checksum-ipv4: off [fixed] 6 tx-checksum-ip-generic: on 7 tx-checksum-ipv6: off [fixed] 8 tx-checksum-fcoe-crc: off [fixed] 9 tx-checksum-sctp: off [fixed] 10scatter-gather: on 11 tx-scatter-gather: on 12 tx-scatter-gather-fraglist: on 13tcp-segmentation-offload: on 14 tx-tcp-segmentation: on 15 tx-tcp-ecn-segmentation: on 16 tx-tcp-mangleid-segmentation: on 17 tx-tcp6-segmentation: on 18generic-segmentation-offload: on 19generic-receive-offload: on 20large-receive-offload: off [fixed] 21rx-vlan-offload: off [fixed] 22tx-vlan-offload: off [fixed] 23ntuple-filters: off [fixed] 24receive-hashing: off [fixed] 25highdma: on 26rx-vlan-filter: off [fixed] 27vlan-challenged: off [fixed] 28tx-lockless: on [fixed] 29netns-local: off [fixed] 30tx-gso-robust: off [fixed] 31tx-fcoe-segmentation: off [fixed] 32tx-gre-segmentation: off [fixed] 33tx-gre-csum-segmentation: off [fixed] 34tx-ipxip4-segmentation: off [fixed] 35tx-ipxip6-segmentation: off [fixed] 36tx-udp_tnl-segmentation: off [fixed] 37tx-udp_tnl-csum-segmentation: off [fixed] 38tx-gso-partial: off [fixed] 39tx-tunnel-remcsum-segmentation: off [fixed] 40tx-sctp-segmentation: on 41tx-esp-segmentation: off [fixed] 42tx-udp-segmentation: on 43tx-gso-list: on 44fcoe-mtu: off [fixed] 45tx-nocache-copy: off 46loopback: off [fixed] 47rx-fcs: off [fixed] 48rx-all: off [fixed] 49tx-vlan-stag-hw-insert: off [fixed] 50rx-vlan-stag-hw-parse: off [fixed] 51rx-vlan-stag-filter: off [fixed] 52l2-fwd-offload: off [fixed] 53hw-tc-offload: off [fixed] 54esp-hw-offload: off [fixed] 55esp-tx-csum-hw-offload: off [fixed] 56rx-udp_tunnel-port-offload: off [fixed] 57tls-hw-tx-offload: off [fixed] 58tls-hw-rx-offload: off [fixed] 59rx-gro-hw: off [fixed] 60tls-hw-record: off [fixed] 61rx-gro-list: off 62macsec-hw-offload: off [fixed] 63rx-udp-gro-forwarding: off 64hsr-tag-ins-offload: off [fixed] 65hsr-tag-rm-offload: off [fixed] 66hsr-fwd-offload: off [fixed] 67hsr-dup-offload: off [fixed]
After disabling them one by one, the issue was solved when TCP segmentation offload (TSO) was disabled. Close to 2 Gbit/s seen now on the IPerf3 test. I was able to enable GSO & GRO again without any issues here.
1fabrizzio@OSR1CR4:~$ ethtool -K tun2 gso off 2fabrizzio@OSR1CR4:~$ ethtool -K tun2 gro off 3fabrizzio@OSR1CR4:~$ ethtool -K tun2 tso off 4 5fabrizzio@OSR1FW2:~$ iperf3 -c 172.27.1.17 6Connecting to host 172.27.1.17, port 5201 7[ 5] local 172.27.1.18 port 57380 connected to 172.27.1.17 port 5201 8[ ID] Interval Transfer Bitrate Retr Cwnd 9[ 5] 0.00-1.00 sec 214 MBytes 1.79 Gbits/sec 370 2.14 MBytes 10[ 5] 1.00-2.00 sec 206 MBytes 1.73 Gbits/sec 0 2.45 MBytes 11[ 5] 2.00-3.00 sec 218 MBytes 1.82 Gbits/sec 121 1.16 MBytes 12[ 5] 3.00-4.00 sec 218 MBytes 1.82 Gbits/sec 0 1.06 MBytes 13[ 5] 4.00-5.00 sec 218 MBytes 1.82 Gbits/sec 0 1.23 MBytes 14[ 5] 5.00-6.00 sec 214 MBytes 1.79 Gbits/sec 202 1.60 MBytes 15[ 5] 6.00-7.00 sec 218 MBytes 1.82 Gbits/sec 471 2.19 MBytes 16[ 5] 7.00-8.00 sec 206 MBytes 1.73 Gbits/sec 383 1.30 MBytes 17[ 5] 8.00-9.00 sec 172 MBytes 1.45 Gbits/sec 0 2.62 MBytes 18[ 5] 9.00-10.00 sec 194 MBytes 1.63 Gbits/sec 0 1.80 MBytes 19- - - - - - - - - - - - - - - - - - - - - - - - - 20[ ID] Interval Transfer Bitrate Retr 21[ 5] 0.00-10.00 sec 2.03 GBytes 1.74 Gbits/sec 1547 sender 22[ 5] 0.00-10.00 sec 2.03 GBytes 1.74 Gbits/sec receiver
Now, there is no option to disable these tunnel offloads via VyOS config. So I'd have to set up a script to run on boot & upon commits to fix this automatically.
But wait, there's more.
So far the troubleshooting has been within GRETAP tunnels that originate & terminate within my private network (that runs IS-IS + SR).
I have also noticed poor behavior on GRE tunnels that ride on top of my private network, but are not originated nor terminated within it. The symptoms were the same, TX drops on the ingress router, on the core-facing interface.
For those, the fix was much easier: Disabling GRO offload on the router ingress interface (not on the core-facing interface).
Changing GRETAP to L2TPv3
I gave this a try - even though that it implies creating a dummy interface to work around VyOS bug T1080
1set interfaces dummy dum3 address '192.168.254.254/24' 2set interfaces dummy dum3 description 'Bug T1080 Workaround' 3 4 5set interfaces l2tpv3 l2tpeth10 description 'OSR1FW2 - OSR1BR1 VLAN 543' 6set interfaces l2tpv3 l2tpeth10 encapsulation 'ip' 7set interfaces l2tpv3 l2tpeth10 mtu '1700' 8set interfaces l2tpv3 l2tpeth10 peer-session-id '543' 9set interfaces l2tpv3 l2tpeth10 peer-tunnel-id '543' 10set interfaces l2tpv3 l2tpeth10 remote '192.168.254.11' 11set interfaces l2tpv3 l2tpeth10 session-id '543' 12set interfaces l2tpv3 l2tpeth10 source-address '192.168.254.13' 13set interfaces l2tpv3 l2tpeth10 tunnel-id '543' 14 15set interfaces bridge br2 member interface l2tpeth10 allowed-vlan '100' 16set interfaces bridge br2 member interface l2tpeth10 native-vlan '100' 17 18 19delete interfaces bridge br2 member interface tun2
It works just fine!. Will have to do this with the rest.
Summary.
In short.
- GRETAP tunnels configured on a VyOS router which sends the traffic encapsulated with MPLS on top of GRE: Either disable TSO on the tunnel with a script, or change the tunnel to L2TPv3.
- GRE tunnels ingressing a VyOS router which then encapsulates again with MPLS: Disable GRO on the router ingress interface.