IPv6 next-hop problem when recursively resolving BGP route
Why can't I SSH to my routers anymore???????
Well, everything started with an upgrade to one of the latest VyOS Nightly builds "vyos-1.4-rolling-202306080317" on some of my AS203528 routers.
1fabrizzio@osr1j1:~$ ping osr2br2 2PING osr2br2(dum0.OSR2BR2.compumundohipermegared.one (2a0e:8f02:21d0:ffff::15)) 56 data bytes 3^C 4--- osr2br2 ping statistics --- 57 packets transmitted, 0 received, 100% packet loss, time 6130ms 6 7fabrizzio@osr1j1:~$ traceroute osr2br2 8traceroute to osr2br2 (2a0e:8f02:21d0:ffff::15), 30 hops max, 80 byte packets 9 1 _gateway (2a0e:8f02:21d1:120::1) 0.247 ms 0.222 ms 0.212 ms 10 2 eth9.osr1cr5.compumundohipermegared.one (2a0e:8f02:21d1:feed:0:1:19:11) 0.423 ms 0.408 ms 0.383 ms 11 3 eth2.osr1cr3.compumundohipermegared.one (2a0e:8f02:21d1:feed:0:1:5:11) 0.774 ms 0.751 ms 0.678 ms 12 4 osr1fw2.compumundohipermegared.one (2a0e:8f02:21d1:ffff::42) 0.860 ms 0.836 ms 0.810 ms 13 5 dum0.OSR1BR2.compumundohipermegared.one (2a0e:8f02:21d0:ffff::13) 1.107 ms 1.079 ms 1.054 ms 14 6 * * * 15 7 * * * 16 8 * * * 17 9 * * * 1810 * * * 1911 *^C
Just by the trace it looked very much like a problem on the reverse path. Luckily I was still able to connect via IPv4. Trying to ping my OSR1 jumphost from OSR2BR2:
1fabrizzio@OSR2BR2:~$ ping 2a0e:8f02:21d1:120::11 2PING 2a0e:8f02:21d1:120::11(2a0e:8f02:21d1:120::11) 56 data bytes 3From 2a0e:8f02:21d0:feed:deed:0:21c:1002 icmp_seq=1 Destination unreachable: Address unreachable 4From 2a0e:8f02:21d0:feed:deed:0:21c:1002 icmp_seq=2 Destination unreachable: Address unreachable 5From 2a0e:8f02:21d0:feed:deed:0:21c:1002 icmp_seq=3 Destination unreachable: Address unreachable 6From 2a0e:8f02:21d0:feed:deed:0:21c:1002 icmp_seq=4 Destination unreachable: Address unreachable 7^C 8--- 2a0e:8f02:21d1:120::11 ping statistics --- 95 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4068ms 10 11fabrizzio@OSR2BR2:~$ sh ipv6 route 2a0e:8f02:21d1:120::11 12Routing entry for 2a0e:8f02:21d1::/48 13 Known via "bgp", distance 200, metric 1000, best 14 Last update 01:11:38 ago 15 fc0e:8f02:21d0:ffff::12 (recursive), weight 1 16 * fe80::7c2a:81ff:fe87:f5a2, via br536, weight 1 17 fc0e:8f02:21d0:ffff::13 (recursive), weight 1 18 * fe80::401d:82ff:fe26:9549, via br540, weight 1
Destination unreachable?
This was very odd for me. I could ping the other end of the tunnel (OSR2BR2 <> OSR1BR1 & OSR2BR2 <> OSR1BR2). Also, the full mesh of IS-IS adjacencies are up at OSR2BR2.
1fabrizzio@OSR2BR2:~$ sh int bridge br536 2br536: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1600 qdisc noqueue state UP group default qlen 1000 3 link/ether c6:21:99:5a:13:d2 brd ff:ff:ff:ff:ff:ff 4 inet6 2a0e:8f02:21d0:feed:deed:0:218:1002/126 scope global 5 valid_lft forever preferred_lft forever 6 inet6 fe80::ac30:5fff:fe1f:b9eb/64 scope link 7 valid_lft forever preferred_lft forever 8 Description: IPv6 Tunnel to OSR1BR1 9 10 RX: bytes packets errors dropped overrun mcast 11 4596611 3846 0 0 0 3825 12 TX: bytes packets errors dropped carrier collisions 13 4774008 4002 0 0 0 0 14fabrizzio@OSR2BR2:~$ sh int bridge br540 15br540: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1600 qdisc noqueue state UP group default qlen 1000 16 link/ether 8e:50:3e:54:f7:01 brd ff:ff:ff:ff:ff:ff 17 inet6 2a0e:8f02:21d0:feed:deed:0:21c:1002/126 scope global 18 valid_lft forever preferred_lft forever 19 inet6 fe80::f01e:3bff:fe29:3d8/64 scope link 20 valid_lft forever preferred_lft forever 21 Description: IPv6 Tunnel to OSR1BR2 22 23 RX: bytes packets errors dropped overrun mcast 24 4641220 3993 0 0 0 3884 25 TX: bytes packets errors dropped carrier collisions 26 4777203 3967 0 0 0 0 27fabrizzio@OSR2BR2:~$ ping 2a0e:8f02:21d0:feed:deed:0:21c:1001 28PING 2a0e:8f02:21d0:feed:deed:0:21c:1001(2a0e:8f02:21d0:feed:deed:0:21c:1001) 56 data bytes 2964 bytes from 2a0e:8f02:21d0:feed:deed:0:21c:1001: icmp_seq=1 ttl=64 time=34.0 ms 3064 bytes from 2a0e:8f02:21d0:feed:deed:0:21c:1001: icmp_seq=2 ttl=64 time=16.9 ms 31^C 32--- 2a0e:8f02:21d0:feed:deed:0:21c:1001 ping statistics --- 332 packets transmitted, 2 received, 0% packet loss, time 1001ms 34rtt min/avg/max/mdev = 16.870/25.443/34.017/8.573 ms 35fabrizzio@OSR2BR2:~$ sh isis neighbor 36Area VyOS: 37 System Id Interface L State Holdtime SNPA 38 AMS1BR1 br533 2 Up 28 2020.2020.2020 39 NYC1BR1 br534 2 Up 29 2020.2020.2020 40 OSR1BR1 br536 2 Up 29 2020.2020.2020 41 OSR1BR2 br540 2 Up 28 2020.2020.2020 42 OSR2BR1 br548 2 Up 29 2020.2020.2020 43 OSR1BR3 br596 2 Up 29 2020.2020.2020 44 OSR2GLASS1 eth8 2 Up 27 2020.2020.2020
Oddly enough I can ping the loopbacks of OSR1BR1 & OSR1BR2 (both GUA and ULA):
1fabrizzio@OSR2BR2:~$ ping 2a0e:8f02:21d0:ffff::12 2PING 2a0e:8f02:21d0:ffff::12(2a0e:8f02:21d0:ffff::12) 56 data bytes 364 bytes from 2a0e:8f02:21d0:ffff::12: icmp_seq=1 ttl=64 time=32.2 ms 4^C 5--- 2a0e:8f02:21d0:ffff::12 ping statistics --- 61 packets transmitted, 1 received, 0% packet loss, time 0ms 7rtt min/avg/max/mdev = 32.191/32.191/32.191/0.000 ms 8 9fabrizzio@OSR2BR2:~$ ping 2a0e:8f02:21d0:ffff::13 10PING 2a0e:8f02:21d0:ffff::13(2a0e:8f02:21d0:ffff::13) 56 data bytes 1164 bytes from 2a0e:8f02:21d0:ffff::13: icmp_seq=1 ttl=64 time=16.9 ms 12^C 13--- 2a0e:8f02:21d0:ffff::13 ping statistics --- 141 packets transmitted, 1 received, 0% packet loss, time 0ms 15rtt min/avg/max/mdev = 16.927/16.927/16.927/0.000 ms 16 17fabrizzio@OSR2BR2:~$ ping fc0e:8f02:21d0:ffff::12 18PING fc0e:8f02:21d0:ffff::12(fc0e:8f02:21d0:ffff::12) 56 data bytes 1964 bytes from fc0e:8f02:21d0:ffff::12: icmp_seq=1 ttl=64 time=16.3 ms 20^C 21--- fc0e:8f02:21d0:ffff::12 ping statistics --- 221 packets transmitted, 1 received, 0% packet loss, time 0ms 23rtt min/avg/max/mdev = 16.340/16.340/16.340/0.000 ms 24 25fabrizzio@OSR2BR2:~$ ping fc0e:8f02:21d0:ffff::13 26PING fc0e:8f02:21d0:ffff::13(fc0e:8f02:21d0:ffff::13) 56 data bytes 2764 bytes from fc0e:8f02:21d0:ffff::13: icmp_seq=1 ttl=64 time=16.2 ms 28^C 29--- fc0e:8f02:21d0:ffff::13 ping statistics --- 301 packets transmitted, 1 received, 0% packet loss, time 0ms 31rtt min/avg/max/mdev = 16.228/16.228/16.228/0.000 ms
Then by comparing the routes:
1fabrizzio@OSR2BR2:~$ sh ipv6 route 2a0e:8f02:21d0:ffff::13 2Routing entry for 2a0e:8f02:21d0:ffff::13/128 3 Known via "isis", distance 115, metric 500, best 4 Last update 01:59:55 ago 5 * fe80::909f:cdff:fe62:592e, via br540, weight 1 <<<<< CORRECT LL address from OSR1BR2 6 7Routing entry for 2a0e:8f02:21d0:ffff::13/128 8 Known via "bgp", distance 200, metric 0 9 Last update 02:00:04 ago 10 fc0e:8f02:21d0:ffff::13 (recursive), weight 1 11 fe80::401d:82ff:fe26:9549, via br540, weight 1 <<<<< where did this come from? 12fabrizzio@OSR2BR2:~$ sh ipv6 route 2a0e:8f02:21d0:ffff::12 13Routing entry for 2a0e:8f02:21d0:ffff::12/128 14 Known via "bgp", distance 200, metric 0 15 Last update 01:16:11 ago 16 fc0e:8f02:21d0:ffff::12 (recursive), weight 1 17 fe80::7c2a:81ff:fe87:f5a2, via br536, weight 1 <<<< where did this come from? 18 19Routing entry for 2a0e:8f02:21d0:ffff::12/128 20 Known via "isis", distance 115, metric 500, best 21 Last update 01:16:11 ago 22 * fe80::60fa:89ff:fe52:4194, via br536, weight 1 <<<<< CORRECT LL address from OSR1BR1 23 24fabrizzio@OSR2BR2:~$ sh ipv6 route fc0e:8f02:21d0:ffff::12 25Routing entry for fc0e:8f02:21d0:ffff::12/128 26 Known via "isis", distance 115, metric 510, best 27 Last update 01:16:22 ago 28 * fe80::60fa:89ff:fe52:4194, via br536, weight 1 <<<<< CORRECT LL address from OSR1BR1 29 30 31sh fabrizzio@OSR2BR2:~$ sh ipv6 route 2a0e:8f02:21d1:120::11 32Routing entry for 2a0e:8f02:21d1::/48 33 Known via "bgp", distance 200, metric 1000, best 34 Last update 01:19:09 ago 35 fc0e:8f02:21d0:ffff::12 (recursive), weight 1 36 * fe80::7c2a:81ff:fe87:f5a2, via br536, weight 1 <<<< Both next hop LL IPs don't match what's on the other end of the tunnel 37 fc0e:8f02:21d0:ffff::13 (recursive), weight 1 38 * fe80::401d:82ff:fe26:9549, via br540, weight 1 <<<< Both next hop LL IPs don't match what's on the other end of the tunnel.
Now here's the issue. For some reason the IPv6 route to the remote router's IPv6 ULA loopback ("fc0e:8f02:21d0:ffff::12", which I am forcing BGP to use) has the correct LL next hop for the other end of the tunnel. But when doing the recursive lookup, as an example towards "2a0e:8f02:21d0:ffff::12/128" or to "2a0e:8f02:21d1::/48", which gets recursively resolved using "fc0e:8f02:21d0:ffff::12", the next-hop found for it is incorrect. I tried changing manually the IPv6 Link-local address on OSR1BR1, the IS-IS route next-hop as seen on OSR2BR2 did change, but when doing the recursive look-up it was stuck on the old link-local address.
1fabrizzio@OSR1BR1:~$ sh interfaces bridge br536 2br536: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1600 qdisc noqueue state UP group default qlen 1000 3 link/ether fe:45:5c:03:c6:ae brd ff:ff:ff:ff:ff:ff 4 inet6 2a0e:8f02:21d0:feed:deed:0:218:1001/126 scope global 5 valid_lft forever preferred_lft forever 6 inet6 fe80::60fa:89ff:fe52:4194/64 scope link <<<<< CORRECT LL address from OSR1BR1 7 valid_lft forever preferred_lft forever 8 Description: IPv6 Tunnel to OSR2BR2 9 10 RX: bytes packets errors dropped overrun mcast 11 2739350 2303 0 0 0 2289 12 TX: bytes packets errors dropped carrier collisions 13 2781627 2287 0 0 0 0 14fabrizzio@OSR1BR2:~$ sh interfaces bridge br540 15br540: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1600 qdisc noqueue state UP group default qlen 1000 16 link/ether 4a:11:ee:ae:f3:87 brd ff:ff:ff:ff:ff:ff 17 inet6 2a0e:8f02:21d0:feed:deed:0:21c:1001/126 scope global 18 valid_lft forever preferred_lft forever 19 inet6 fe80::909f:cdff:fe62:592e/64 scope link <<<<< CORRECT LL address from OSR1BR2 20 valid_lft forever preferred_lft forever 21 Description: IPv6 Tunnel to OSR2BR2 22 23 RX: bytes packets errors dropped overrun mcast 24 4288601 3568 0 0 0 3552 25 TX: bytes packets errors dropped carrier collisions" 26 4356271 3683 0 0 0 0
I truly have no idea what might be going on here. This one is an example. I had the issue occur on other routers and rebooting them was playing a nasty whack-a-mole game with the issue appearing elsewhere. Clearing BGP neighbors didn't fix this issue as well. The thing is that the MAC addresses assigned to the tunnels change upon a router reboot. Therefore if you reboot router A then all the other tunnels from the other routers (with the suspected software bug) pointing towards A will still have the old IPv6 link-local next hop of the tunnel endpoints at A.
I've just rolled back to known good version "1.4-rolling-202210280218" for now. If I get some spare time I will lab this up and file a bug with VyOS. I don't know if it's VyOS bug or FRR bug to be fair. "1.4-rolling-202210280218" uses FRR 8.3.1, the nightly I tried was "vyos-1.4-rolling-202306080317" with FRR 8.5.1
Hope this helps someone.