VyOS and BGP MED
Inefficient Routing
An important part of my (new) network is the integration with my old Linodes, on which I rely for the IPv4/v6 addressing for my authoritative nameservers, among other things.
That's connected to my current internal network through a variety of tunnels (That's a discussion for another day).
Current setup
The problem I have been facing is with the return traffic from my Linodes to my internal network. That (new) internal network spans two different physical locations, however it is completely integrated (single private ASN / single IGP, both sites connected via GRETAP over AS203528).
I would like to have the routers in OSR1 / OSR2 advertise via BGP the prefixes of the (OSR1+OSR2) network, to the old Linodes, using MED value of the IGP metric to reach the iBGP next hops. This would be something like Juniper's "metric-out igp" BGP config. Unfortunately I have not found a VyOS config parameter to do that. The most you can do is set up a route-map to force an specific MED value.
Default behavior: no MED sent.
1fabrizzio@OSR1CR2:~$ sh ip bgp ipv4 unicast neighbors 192.168.251.209 advertised-routes | match "Metric|192.168.20.0/24" 2 Network Next Hop Metric LocPrf Weight Path 3*> 192.168.20.0/24 0.0.0.0 100 0 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 i
Because of this, unless I do something to amend this situation, traffic from either towards my network can be sent to either OSR1 or OSR2 (if it arrives at the wrong POP it must be "backhauled"). Those routes will be equivalent.
Dirty Fix
1192.168.251.186 4 4200000001 200881 153094 0 0 0 01:14:05 109 121 To OSR2CR2 2192.168.251.194 4 4200000001 175286 143443 0 0 0 01:14:04 109 121 To OSR1CR1 3 4fabrizzio@FFT1EV1:~$ sh ip bgp ipv4 unicast neighbors 192.168.251.186 routes | match "Metric|192.168.20.0/24" 5 Network Next Hop Metric LocPrf Weight Path 6* 192.168.20.0/24 192.168.251.186 10000 0 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 i 7fabrizzio@FFT1EV1:~$ sh ip bgp ipv4 unicast neighbors 192.168.251.194 routes | match "Metric|192.168.20.0/24" 8 Network Next Hop Metric LocPrf Weight Path 9* 192.168.20.0/24 192.168.251.194 0 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 i 10000001 4200000001 i 11 12set policy route-map METRIC_10000 rule 10 action 'permit' 13set policy route-map METRIC_10000 rule 10 set metric '10000' 14 15set protocols bgp neighbor 192.168.251.186 address-family ipv4-unicast route-map import 'METRIC_10000' 16set protocols bgp neighbor 192.168.251.186 address-family ipv6-unicast route-map import 'METRIC_10000'
I have more capacity at OSR1, therefore I forced a MED value on the BGP peerings from each Linode to OSR2 nodes. Traffic destined to OSR2 will route via OSR1 (then the GRETAP to OSR2). Outbound traffic from OSR2 will be direct.
To be fair this works just fine. I could mess with the route map to make it filter some prefixes, or maybe two route maps (to prefer OSR1 prefixes on OSR1 tunnel, and OSR2 prefixes on OSR2 tunnel).
However this can be made better, much better.
Communities to the rescue
Easiest way would be having the route reflectors within my internal network, add a community on the routes coming from each RR-client, based on the node location.
1set policy route-map RTR_OSR1 rule 10 action 'permit' 2set policy route-map RTR_OSR1 rule 10 set community add '65000:101' 3set policy route-map RTR_OSR2 rule 10 action 'permit' 4set policy route-map RTR_OSR2 rule 10 set community add '65000:102' 5 6 7 8set protocols bgp neighbor 192.168.254.10 address-family ipv4-unicast route-map import RTR_OSR1 9set protocols bgp neighbor 192.168.254.10 address-family ipv6-unicast route-map import RTR_OSR1 10 11 12set protocols bgp neighbor 192.168.254.16 address-family ipv4-unicast route-map import RTR_OSR2 13set protocols bgp neighbor 192.168.254.16 address-family ipv6-unicast route-map import RTR_OSR2
This works just fine: (I have four RRs)
1fabrizzio@OSR1CR1:~$ sh ip bgp ipv4 unicast 192.168.20.0/24 2BGP routing table entry for 192.168.20.0/24, version 128126 3Paths: (4 available, best #4, table default) 4 Local 5 192.168.254.33 (metric 1110) from 192.168.254.53 (192.168.254.33) 6 Origin IGP, metric 0, localpref 100, valid, internal 7 Community: 65000:101 8 Originator: 192.168.254.33, Cluster list: 4.4.4.4 9 AddPath ID: RX 85, TX-All 1231 TX-Best-Per-AS 0 10 Last update: Fri Mar 3 18:07:12 2023 11 Local 12 192.168.254.33 (metric 1110) from 192.168.254.51 (192.168.254.33) 13 Origin IGP, metric 0, localpref 100, valid, internal 14 Community: 65000:101 15 Originator: 192.168.254.33, Cluster list: 2.2.2.2 16 AddPath ID: RX 477, TX-All 1724 TX-Best-Per-AS 0 17 Last update: Fri Mar 3 18:06:24 2023 18 Local 19 192.168.254.33 (metric 1110) from 192.168.254.52 (192.168.254.33) 20 Origin IGP, metric 0, localpref 100, valid, internal 21 Community: 65000:101 22 Originator: 192.168.254.33, Cluster list: 3.3.3.3 23 AddPath ID: RX 478, TX-All 1728 TX-Best-Per-AS 0 24 Last update: Fri Mar 3 18:06:50 2023 25 Local 26 192.168.254.33 (metric 1110) from 192.168.254.50 (192.168.254.33) 27 Origin IGP, metric 0, localpref 100, valid, internal, best (Neighbor IP) 28 Community: 65000:101 29 Originator: 192.168.254.33, Cluster list: 1.1.1.1 30 AddPath ID: RX 199, TX-All 1729 TX-Best-Per-AS 0 31 Advertised to: 172.27.18.42 172.27.18.58 192.168.251.193 32 Last update: Fri Mar 3 18:05:05 2023 33fabrizzio@OSR1CR1:~$ sh ip bgp ipv4 unicast 192.168.35.0/24 34BGP routing table entry for 192.168.35.0/24, version 128070 35Paths: (4 available, best #3, table default) 36 Local 37 192.168.254.34 (metric 45110) from 192.168.254.53 (192.168.254.34) 38 Origin IGP, metric 0, localpref 100, valid, internal 39 Community: 65000:102 40 Originator: 192.168.254.34, Cluster list: 4.4.4.4 41 AddPath ID: RX 4, TX-All 1005 TX-Best-Per-AS 0 42 Last update: Fri Mar 3 18:07:12 2023 43 Local 44 192.168.254.34 (metric 45110) from 192.168.254.51 (192.168.254.34) 45 Origin IGP, metric 0, localpref 100, valid, internal 46 Community: 65000:102 47 Originator: 192.168.254.34, Cluster list: 2.2.2.2 48 AddPath ID: RX 9, TX-All 393 TX-Best-Per-AS 0 49 Last update: Fri Mar 3 18:06:24 2023 50 Local 51 192.168.254.34 (metric 45110) from 192.168.254.50 (192.168.254.34) 52 Origin IGP, metric 0, localpref 100, valid, internal, best (Neighbor IP) 53 Community: 65000:102 54 Originator: 192.168.254.34, Cluster list: 1.1.1.1 55 AddPath ID: RX 88, TX-All 396 TX-Best-Per-AS 0 56 Advertised to: 172.27.18.42 172.27.18.58 192.168.251.193 57 Last update: Fri Mar 3 18:05:05 2023 58 Local 59 192.168.254.34 (metric 45110) from 192.168.254.52 (192.168.254.34) 60 Origin IGP, metric 0, localpref 100, valid, internal 61 Community: 65000:102 62 Originator: 192.168.254.34, Cluster list: 3.3.3.3 63 AddPath ID: RX 27, TX-All 400 TX-Best-Per-AS 0 64 Last update: Fri Mar 3 18:06:50 2023
Then on the routers at OSR1 / OSR2 peering with my Linodes:
1set policy community-list FROM_OSR1 rule 10 action 'permit' 2set policy community-list FROM_OSR1 rule 10 regex '65000:101' 3set policy community-list FROM_OSR2 rule 10 action 'permit' 4set policy community-list FROM_OSR2 rule 10 regex '65000:102' 5 6set policy route-map LINODE_MED_FROM_OSR1 rule 10 action 'permit' 7set policy route-map LINODE_MED_FROM_OSR1 rule 10 match community community-list 'FROM_OSR1' 8set policy route-map LINODE_MED_FROM_OSR1 rule 10 set metric '100' 9set policy route-map LINODE_MED_FROM_OSR1 rule 20 action 'permit' 10set policy route-map LINODE_MED_FROM_OSR1 rule 20 match community community-list 'FROM_OSR2' 11set policy route-map LINODE_MED_FROM_OSR1 rule 20 set metric '200' 12 13 14set policy route-map PREPEND-9-LINODE rule 10 action 'permit' 15set policy route-map PREPEND-9-LINODE rule 10 set as-path prepend '4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001' 16set policy route-map PREPEND-9-LINODE rule 10 call 'LINODE_MED_FROM_OSR1' 17 18set protocols bgp neighbor 192.168.251.209 address-family ipv4-unicast route-map export PREPEND-9-LINODE 19set protocols bgp neighbor 192.168.251.209 address-family ipv6-unicast route-map export PREPEND-9-LINODE
(The AS prepending will vary depending on the distance, I ran BGP as IGP before and this is just a remnant).
Results
1fabrizzio@OSR1CR2:~$ sh ip bgp ipv4 unicast neighbors 192.168.251.209 advertised-routes | match "Metric|192.168.20.0/24" 2 Network Next Hop Metric LocPrf Weight Path 3*> 192.168.20.0/24 0.0.0.0 100 100 0 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 i 4fabrizzio@OSR1CR2:~$ sh ip bgp ipv4 unicast neighbors 192.168.251.209 advertised-routes | match "Metric|192.168.35.0/24" 5 Network Next Hop Metric LocPrf Weight Path 6*> 192.168.35.0/24 0.0.0.0 200 100 0 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 i
Now the metrics for the prefixes depend on the site. This one was easy :D
The mistake and fix
After deploying this I was trying to figure out why some routes were not being advertised.
1172.27.18.58 4 65108 1130276 1166833 0 0 0 01w6d08h 2 121 To OSR1E3 2192.168.251.193 4 65007 47890 55758 0 0 0 00:27:11 36 106 To FFT1EV1 *LEGACY* 3 4 5fabrizzio@OSR1CR1:~$ sh ip bgp ipv4 unicast neighbors 172.27.18.58 routes 6BGP table version is 128213, local router ID is 192.168.254.10, vrf id 0 7Default local pref 100, local AS 4200000001 8Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, 9 i internal, r RIB-failure, S Stale, R Removed 10Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self 11Origin codes: i - IGP, e - EGP, ? - incomplete 12RPKI validation codes: V valid, I invalid, N Not found 13 14 Network Next Hop Metric LocPrf Weight Path 15*> 192.168.39.0/24 172.27.18.58 0 0 65108 i 16*> 192.168.45.128/26 17 172.27.18.58 0 0 65108 i 18 19Displayed 2 routes and 1787 total paths 20fabrizzio@OSR1CR1:~$ sh ip bgp ipv4 unicast neighbors 192.168.251.193 advertised-routes | match "192.168.39.0/24" 21fabrizzio@OSR1CR1:~$
The core routers also have eBGP sessions to routers that were also part of my legacy network (or just the Mikrotik boxes which I don't want on my IGP!). These routes that are received via eBGP (or just redistributed into BGP) instead of coming from a route reflector, don't have the OSR1/OSR2 community - therefore are dropped.
1fabrizzio@OSR1CR1:~$ sh ip bgp ipv4 unicast 192.168.39.0/24 2BGP routing table entry for 192.168.39.0/24, version 128102 3Paths: (5 available, best #5, table default) 4<snip> 5 65108 6 172.27.18.58 from 172.27.18.58 (192.168.45.129) 7 Origin IGP, metric 0, valid, external, best (AS Path) 8 AddPath ID: RX 0, TX-All 9 TX-Best-Per-AS 0 9 Advertised to: 172.27.18.42 172.27.18.58 192.168.254.50 192.168.254.51 192.168.254.52 192.168.254.53 10 Last update: Sat Feb 18 11:18:30 2023
Summary of the issue
Fix and results after:
1set policy route-map LINODE_MED_FROM_OSR1 rule 10 action 'permit' 2set policy route-map LINODE_MED_FROM_OSR1 rule 10 match community community-list 'FROM_OSR1' 3set policy route-map LINODE_MED_FROM_OSR1 rule 10 set metric '100' 4set policy route-map LINODE_MED_FROM_OSR1 rule 20 action 'permit' 5set policy route-map LINODE_MED_FROM_OSR1 rule 20 match community community-list 'FROM_OSR2' 6set policy route-map LINODE_MED_FROM_OSR1 rule 20 set metric '200' 7set policy route-map LINODE_MED_FROM_OSR1 rule 30 action permit 8set policy route-map LINODE_MED_FROM_OSR1 rule 30 set metric '100' 9set policy route-map LINODE_MED_FROM_OSR1 rule 30 description 'Catch-all for local non-reflected routes' 10 11 12fabrizzio@OSR1CR1:~$ sh ip bgp ipv4 unicast neighbors 192.168.251.193 advertised-routes | match "192.168.39.0/24" 13*> 192.168.39.0/24 0.0.0.0 100 0 4200000001 4200000001 4200000001 4200000001 4200000001 4200000001 65108 i
Was an easy thing to fix at least :D