VyOS 1.5, EVPN-VXLAN First test on VyOS
Today I wanted to play around with VXLAN as I've never touched it and I had nothing better to do.
This has been tested with the latest VyOS nightly 1.5-rolling-202403050022. First, I read the related VyOS blog post and the FRR docs. I really prefer learning while doing things instead of just reading the theory. Even though it means I will likely mess something up, that's precisely what the Lab is there for.
EVPN - BGP
First I need to enable address-familty l2vpn-evpn on my route reflectors. I have four of them, I will only be touching one for now (They all have the same copy of same routes).
I will change the L2TPv3 PW I have internally between OSR1CR1 to OSR1CR3 (I use it for something equivalent to a EVPL, internal backhaul of my WAN from one server to another), to EVPN-VXLAN. These two routers are adjacent to each other, connected via 2x2.5GbE. My Lab has IS-IS as an IGP, MPLS-enabled via SR.
Example config at OSR1RR1 (Route Reflector) for iBGP RR-client peering to OSR1CR3 (Core):
1fabrizzio@OSR1RR1# show protocols bgp neighbor 192.168.254.12 2 address-family { 3 ipv4-unicast { 4 addpath-tx-all 5 route-map { 6 import RTR_OSR1 7 } 8 route-reflector-client 9 } 10 ipv6-unicast { 11 addpath-tx-all 12 route-map { 13 import RTR_OSR1 14 } 15 route-reflector-client 16 } 17 } 18 bfd { 19 profile IBGP_BFD 20 } 21 description OSR1CR3 22 remote-as internal 23 timers { 24 connect 1 25 } 26 update-source dum0
I just need a couple of commands on the RR:
1fabrizzio@OSR1RR1# set protocols bgp neighbor 192.168.254.12 address-family l2vpn-evpn route-reflector-client 2fabrizzio@OSR1RR1# set protocols bgp neighbor 192.168.254.14 address-family l2vpn-evpn route-reflector-client 3 4 5L2VPN EVPN Summary (VRF default): 6BGP router identifier 192.168.254.50, local AS number 4200000001 vrf-id 0 7BGP table version 0 8RIB entries 0, using 0 bytes of memory 9Peers 2, using 40 KiB of memory 10 11Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc 12192.168.254.12 4 4200000001 39 265 0 0 0 00:00:26 NoNeg NoNeg OSR1CR3 13192.168.254.14 4 4200000001 50 267 0 0 0 00:00:10 NoNeg NoNeg OSR1CR5
then on the core routers:
1neighbor 192.168.254.50 { 2 address-family { 3 ipv4-unicast { 4 addpath-tx-all 5 nexthop-self { 6 } 7 route-map { 8 import prevent_ibgp_blackholing 9 } 10 soft-reconfiguration { 11 inbound 12 } 13 } 14 ipv6-unicast { 15 addpath-tx-all 16 nexthop-self { 17 } 18 route-map { 19 export ibgp_ula_nh 20 import prevent_ibgp_blackholing 21 } 22 soft-reconfiguration { 23 inbound 24 } 25 } 26 } 27 bfd { 28 profile IBGP_BFD 29 } 30 description "To OSR1RR1" 31 remote-as internal 32 update-source dum0 33 34 35fabrizzio@OSR1CR3# set protocols bgp neighbor 192.168.254.50 address-family l2vpn-evpn nexthop-self 36 37fabrizzio@OSR1CR3# set protocols bgp address-family l2vpn-evpn advertise-all-vni
This should be enough BGP for today :) I did the same on OSR1CR5.
Moving from L2TPv3 tunnel to VXLAN
Switching over is easy. First I deploy VXLAN interface on one of the core routers
1fabrizzio@OSR1CR3# set interfaces vxlan vxlan700 parameters nolearning 2fabrizzio@OSR1CR3# set interfaces vxlan vxlan700 port 4789 3fabrizzio@OSR1CR3# set interfaces vxlan vxlan700 source-address 192.168.254.12 4fabrizzio@OSR1CR3# set interfaces vxlan vxlan700 vni 700
Then just switch over one bridge member from L2TPv3 to VXLAN
1fabrizzio@OSR1CR3# show interfaces bridge br5 2 description "WAN OSR1BR2 - OSR1CR3 - OSR1CR5 VLAN 700 BR2" 3 enable-vlan 4 ipv6 { 5 address { 6 no-default-link-local 7 } 8 } 9 member { 10 interface eth17 { 11 allowed-vlan 100 12 native-vlan 100 13 } 14 interface l2tpeth5 { 15 allowed-vlan 100 16 native-vlan 100 17 } 18 } 19 20fabrizzio@OSR1CR3# delete interfaces bridge br5 member interface l2tpeth5 21fabrizzio@OSR1CR3# set interfaces bridge br5 member interface vxlan700 allowed-vlan 100 22fabrizzio@OSR1CR3# set interfaces bridge br5 member interface vxlan700 native-vlan 100
MACs are already being sent via BGP-EVPN at this point
1fabrizzio@OSR1CR3:~$ show bgp l2vpn evpn 2BGP table version is 2, local router ID is 192.168.254.12 3Status codes: s suppressed, d damped, h history, * valid, > best, i - internal 4Origin codes: i - IGP, e - EGP, ? - incomplete 5EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id] 6EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP] 7EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP] 8EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP] 9EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP] 10 11 Network Next Hop Metric LocPrf Weight Path 12Route Distinguisher: 192.168.254.12:2 13 *> [2]:[0]:[48]:[ce:a7:7a:xx:xx:xx] 14 192.168.254.12 32768 i 15 ET:8 RT:59905:700 16 *> [3]:[0]:[32]:[192.168.254.12] 17 192.168.254.12 32768 i 18 ET:8 RT:59905:700
After switching over from L2TPv3 to VXLAN on OSR1CR5 there are MACs learnt and sent over BGP at both ends:
1fabrizzio@OSR1CR5:~$ show bgp l2vpn evpn 2BGP table version is 2, local router ID is 192.168.254.14 3Status codes: s suppressed, d damped, h history, * valid, > best, i - internal 4Origin codes: i - IGP, e - EGP, ? - incomplete 5EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id] 6EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP] 7EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP] 8EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP] 9EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP] 10 11 Network Next Hop Metric LocPrf Weight Path 12Route Distinguisher: 192.168.254.12:2 13 *>i[2]:[0]:[48]:[ce:a7:7a:xx:xx:xx] 14 192.168.254.12 0 100 0 i 15 RT:59905:700 ET:8 16 *>i[3]:[0]:[32]:[192.168.254.12] 17 192.168.254.12 0 100 0 i 18 RT:59905:700 ET:8 19Route Distinguisher: 192.168.254.14:2 20 *> [2]:[0]:[48]:[6c:eb:b6:xx:xx:xx] 21 192.168.254.14 32768 i 22 ET:8 RT:59905:700 23 *> [3]:[0]:[32]:[192.168.254.14] 24 192.168.254.14 32768 i 25 ET:8 RT:59905:700
I also made sure that it's not reordering traffic within a flow. OSR1CR3 to OSR1CR5 are adjacent to each other, connected by 2x2.5GbE links, I ran a 16-thread IPerf3 test over the VXLAN-L2 connection and there was no reordering within the same flow. Furthermore the ECMP paths OSR1CR3 <> OSR1CR5 were both utilized!!
Because I am running IS-IS + MPLS (SR) internally, I wanted to make sure that there wouldn't be any problems with VXLAN if the IPv4 next-hop would have an MPLS label. The test OSR1CR3 <> OSR1CR5 doesn't really have any labels (adjacent to each other w/PHP, so implicit null)
1fabrizzio@OSR1CR3:~$ sh ip route 192.168.254.14 2Routing entry for 192.168.254.14/32 3 Known via "isis", distance 115, metric 1010, best 4 Last update 04:03:32 ago 5 * 172.27.16.18, via eth2, label implicit-null, weight 1 6 * 172.27.16.22, via eth3, label implicit-null, weight 1
I also haven't bothered trying this out in an actual multi-point fashion so far.
So the best way to try this out would be testing this again, this time creating a bridge interface at OSR2CR2 (at OSR2) connected to a VLAN there. Then bringing it to OSR1CR6 to a bridge there. This is not totally useless as I can bring one of my LANs from OSR2 to OSR1 in case I need to troubleshoot anything. OSR1CR6 and OSR2CR2 are not adjacent to each other so there will be MPLS labels involved here.
1fabrizzio@OSR1CR6:~$ sh ip route 192.168.254.17 2Routing entry for 192.168.254.17/32 3 Known via "isis", distance 115, metric 46110, best 4 Last update 04:45:11 ago 5 * 172.27.16.45, via eth0, label 16170, weight 1
Performance issues
After doing the configs I ran into exactly the same performance issue with TX drops on the core facing interface at the edge routers and large TCP segments being dropped :( Same thing I saw on my previous post.
1fabrizzio@OSR1CR6:~$ sh interfaces ethernet eth0 2eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1800 qdisc mq state UP group default qlen 1000 3 link/ether 52:d6:dc:f3:9c:28 brd ff:ff:ff:ff:ff:ff 4 altname enp0s18 5 altname ens18 6 inet 172.27.16.46/30 brd 172.27.16.47 scope global eth0 7 valid_lft forever preferred_lft forever 8 inet6 2a0e:8f02:21d1:feed:0:1:12:12/126 scope global 9 valid_lft forever preferred_lft forever 10 inet6 fe80::50d6:dcff:fef3:9c28/64 scope link 11 valid_lft forever preferred_lft forever 12 Description: To OSR1CR5 13 14 RX: bytes packets errors dropped overrun mcast 15 126316674 369467 0 8 0 0 16 TX: bytes packets errors dropped carrier collisions 17 140990639 833966 0 4944 0 0 <<<< 18 19 fabrizzio@osr1test3:~$ iperf3 -c 192.168.35.3 20Connecting to host 192.168.35.3, port 5201 21[ 5] local 192.168.35.109 port 48946 connected to 192.168.35.3 port 5201 22[ ID] Interval Transfer Bitrate Retr Cwnd 23[ 5] 0.00-1.00 sec 160 KBytes 1.31 Mbits/sec 32 4.24 KBytes 24[ 5] 1.00-2.00 sec 80.6 KBytes 660 Kbits/sec 18 4.24 KBytes 25[ 5] 2.00-3.00 sec 119 KBytes 973 Kbits/sec 18 7.07 KBytes 26[ 5] 3.00-4.00 sec 79.2 KBytes 649 Kbits/sec 24 2.83 KBytes 27[ 5] 4.00-5.00 sec 79.2 KBytes 649 Kbits/sec 18 5.66 KBytes 28[ 5] 5.00-6.00 sec 79.2 KBytes 649 Kbits/sec 18 5.66 KBytes 29[ 5] 6.00-7.00 sec 119 KBytes 973 Kbits/sec 30 2.83 KBytes 30[ 5] 7.00-8.00 sec 79.2 KBytes 649 Kbits/sec 20 1.41 KBytes 31[ 5] 8.00-9.00 sec 39.6 KBytes 324 Kbits/sec 14 4.24 KBytes 32[ 5] 9.00-10.00 sec 119 KBytes 974 Kbits/sec 24 2.83 KBytes 33- - - - - - - - - - - - - - - - - - - - - - - - - 34[ ID] Interval Transfer Bitrate Retr 35[ 5] 0.00-10.00 sec 953 KBytes 781 Kbits/sec 216 sender 36[ 5] 0.00-10.02 sec 872 KBytes 714 Kbits/sec receiver
Because I really really want EVPN+VXLAN to be working, I will just assign each router another loopback IP in addition to their current one. I will just not assign them any prefix SID under segment routing so I don't get any MPLS goodness on them.
1fabrizzio@OSR1CR6# set interfaces dummy dum4 address 192.168.254.117/32 2fabrizzio@OSR1CR6# set interfaces dummy dum4 description "For VXLAN - no MPLS/SR" 3fabrizzio@OSR1CR6# set protocols isis interface dum4 passive 4fabrizzio@OSR1CR6# set interfaces vxlan vxlan35 source-address 192.168.254.115
That didn't fix the issue
1fabrizzio@osr1test3:~$ iperf3 -c 192.168.35.3 -P4 -R 2Connecting to host 192.168.35.3, port 5201 3Reverse mode, remote host 192.168.35.3 is sending 4[ 5] local 192.168.35.109 port 42122 connected to 192.168.35.3 port 5201 5[ 7] local 192.168.35.109 port 42130 connected to 192.168.35.3 port 5201 6[ 13] local 192.168.35.109 port 42140 connected to 192.168.35.3 port 5201 7[ 15] local 192.168.35.109 port 42152 connected to 192.168.35.3 port 5201 8^C[ ID] Interval Transfer Bitrate 9[ 5] 0.00-0.83 sec 62.2 KBytes 616 Kbits/sec 10[ 7] 0.00-0.83 sec 65.0 KBytes 644 Kbits/sec 11[ 13] 0.00-0.83 sec 42.4 KBytes 420 Kbits/sec 12[ 15] 0.00-0.83 sec 45.2 KBytes 448 Kbits/sec 13[SUM] 0.00-0.83 sec 215 KBytes 2.13 Mbits/sec 14- - - - - - - - - - - - - - - - - - - - - - - - - 15[ ID] Interval Transfer Bitrate 16[ 5] 0.00-0.83 sec 0.00 Bytes 0.00 bits/sec sender 17[ 5] 0.00-0.83 sec 62.2 KBytes 616 Kbits/sec receiver 18[ 7] 0.00-0.83 sec 0.00 Bytes 0.00 bits/sec sender 19[ 7] 0.00-0.83 sec 65.0 KBytes 644 Kbits/sec receiver 20[ 13] 0.00-0.83 sec 0.00 Bytes 0.00 bits/sec sender 21[ 13] 0.00-0.83 sec 42.4 KBytes 420 Kbits/sec receiver 22[ 15] 0.00-0.83 sec 0.00 Bytes 0.00 bits/sec sender 23[ 15] 0.00-0.83 sec 45.2 KBytes 448 Kbits/sec receiver 24[SUM] 0.00-0.83 sec 0.00 Bytes 0.00 bits/sec sender 25[SUM] 0.00-0.83 sec 215 KBytes 2.13 Mbits/sec receiver 26iperf3: interrupt - the client has terminated 27 28 29 30fabrizzio@OSR1CR6:~$ sh int ethernet eth0 31eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1800 qdisc mq state UP group default qlen 1000 32 link/ether 52:d6:dc:f3:9c:28 brd ff:ff:ff:ff:ff:ff 33 altname enp0s18 34 altname ens18 35 inet 172.27.16.46/30 brd 172.27.16.47 scope global eth0 36 valid_lft forever preferred_lft forever 37 inet6 2a0e:8f02:21d1:feed:0:1:12:12/126 scope global 38 valid_lft forever preferred_lft forever 39 inet6 fe80::50d6:dcff:fef3:9c28/64 scope link 40 valid_lft forever preferred_lft forever 41 Description: To OSR1CR5 42 43 RX: bytes packets errors dropped overrun mcast 44 3152902 5949 0 25 0 0 45 TX: bytes packets errors dropped carrier collisions 46 2976484 6099 0 284 0 0 <<<<
Just as the last time, disabling TSO on the tunnel interface (for this case VXLAN35 on both OSR1CR6 and OSR2CR2) fixes the problem:
1fabrizzio@OSR1CR6:~$ ethtool -K vxlan35 tso off 2 3 4fabrizzio@osr1test3:~$ iperf3 -c 192.168.35.3 5Connecting to host 192.168.35.3, port 5201 6[ 5] local 192.168.35.109 port 47178 connected to 192.168.35.3 port 5201 7[ ID] Interval Transfer Bitrate Retr Cwnd 8[ 5] 0.00-1.00 sec 33.3 MBytes 280 Mbits/sec 204 573 KBytes 9[ 5] 1.00-2.00 sec 27.5 MBytes 231 Mbits/sec 0 615 KBytes 10[ 5] 2.00-3.00 sec 28.8 MBytes 241 Mbits/sec 0 642 KBytes 11[ 5] 3.00-4.00 sec 25.0 MBytes 210 Mbits/sec 21 662 KBytes 12[ 5] 4.00-5.00 sec 28.8 MBytes 241 Mbits/sec 8 509 KBytes 13[ 5] 5.00-6.00 sec 27.5 MBytes 231 Mbits/sec 0 576 KBytes 14[ 5] 6.00-7.00 sec 31.2 MBytes 262 Mbits/sec 0 624 KBytes 15[ 5] 7.00-8.00 sec 30.0 MBytes 252 Mbits/sec 0 655 KBytes 16[ 5] 8.00-9.00 sec 32.5 MBytes 273 Mbits/sec 0 672 KBytes 17[ 5] 9.00-10.00 sec 26.2 MBytes 220 Mbits/sec 130 495 KBytes 18- - - - - - - - - - - - - - - - - - - - - - - - - 19[ ID] Interval Transfer Bitrate Retr 20[ 5] 0.00-10.00 sec 291 MBytes 244 Mbits/sec 363 sender 21[ 5] 0.00-10.02 sec 288 MBytes 242 Mbits/sec receiver
Permanent fix
I asked ChatGPT to give me a bash script to disable TSO on all VXLAN interfaces:
1Hi dear bot. 2I want to run a bash script that will do 3ethtool -K [interface_name] tso off 4for all interfaces starting with "vxlan". 5could you please let me know how to do it?. I promise I will tip you $200 :)
ChatGPT gave me:
1#!/bin/bash 2 3# Get a list of all network interfaces 4interfaces=$(ls /sys/class/net) 5 6# Loop through each interface 7for interface in $interfaces; do 8 # Check if the interface name starts with "vxlan" 9 if [[ $interface == vxlan* ]]; then 10 # Disable TSO for the interface 11 sudo ethtool -K $interface tso off 12 echo "Disabled TSO for interface $interface" 13 fi 14done
Works wonders :)
I rolled back the new loopbacks, the EVPN-VXLAN works just fine even though it is running over MPLS - no need to complicate my config with new loopbacks.
120:03:55.585321 MPLS (label 16170, tc 0, [S], ttl 16) IP 192.168.254.15.34338 > 192.168.254.17.4789: VXLAN, flags [I] (0x08), vni 35 2IP 192.168.35.109.38590 > 192.168.35.3.5201: Flags [.], seq 35796045:35797493, ack 1, win 502, options [nop,nop,TS val 203409691 ecr 118461559], length 1448 320:03:55.585322 MPLS (label 16170, tc 0, [S], ttl 16) IP 192.168.254.15.34338 > 192.168.254.17.4789: VXLAN, flags [I] (0x08), vni 35 4IP 192.168.35.109.38590 > 192.168.35.3.5201: Flags [.], seq 35797493:35798941, ack 1, win 502, options [nop,nop,TS val 203409691 ecr 118461559], length 1448 520:03:55.585322 MPLS (label 16170, tc 0, [S], ttl 16) IP 192.168.254.15.34338 > 192.168.254.17.4789: VXLAN, flags [I] (0x08), vni 35 6IP 192.168.35.109.38590 > 192.168.35.3.5201: Flags [.], seq 35798941:35800389, ack 1, win 502, options [nop,nop,TS val 203409691 ecr 118461559], length 1448 720:03:55.585343 MPLS (label 16170, tc 0, [S], ttl 16) IP 192.168.254.15.34338 > 192.168.254.17.4789: VXLAN, flags [I] (0x08), vni 35
I've then deployed this script to run on commit and on boot after config:
1echo '# Get a list of all network interfaces 2interfaces=$(ls /sys/class/net) 3 4# Loop through each interface 5for interface in $interfaces; do 6 # Check if the interface name starts with "vxlan" 7 if [[ $interface == vxlan* ]]; then 8 # Disable TSO for the interface 9 sudo ethtool -K $interface tso off 10 echo "Disabled TSO for interface $interface" 11 fi 12done' >> /config/scripts/vyos-postconfig-bootup.script 13mkdir /config/scripts/commit 14mkdir /config/scripts/commit/post-hooks.d 15cp /config/scripts/vyos-postconfig-bootup.script /config/scripts/commit/post-hooks.d/98-disable-tso-on-vxlan.script
All is good now :)