AS203528 migration to (shared) route reflectors
My hobby network (AS203528) keeps growing. I had a full iBGP mesh among those routers, however that’s no longer viable.
I already have 4xBGP Route reflectors on my internal network. I want to re-use those same reflectors for AS203528. Those two networks have each their own IGP, and are just peered with each other.
The Plan
My entire network needs to be able to pass a cold start / bootstrap itself. I have the loopbacks of my internal network on 192.168.254.0/24 and of AS203528 at 192.168.248.0/24. Even though each loopback subnet is a /32, only the /24 crosses network boundaries.
I don’t think that it’s a good idea for the connectivity from AS203528 to/from its route reflectors to depend on this summarized prefix. Instead I want the /32 loopbacks to cross networks, to prevent a network split or router isolation to cause major impact.
Obviously the main point is that AS203528 currently knows 192.168.254.0/24 via BGP. The AS203528 border routers that peer with the private network know it via eBGP and the rest of AS203528 via iBGP. This can’t be the case anymore, otherwise how can the network bootstrap itself if it needs BGP to reach its route reflectors?.
Therefore I am going to distribute the individual /32s from 192.168.254.0/24 from IS-IS into BGP on my private network - then send them to AS203528 via eBGP, and then redistribute them from BGP into OSPF within AS203528. This way the bootstrap will work.
I apologize for the crappy diagrams.
This diagram will explain how the route reflector loopbacks will flow from private into AS203528:
And how the loopbacks from AS203528 flow into the private network:
Steps
First I started by redistributing the required prefixes from IS-IS into BGP at the Firewall/Routers on the Private side. I just make sure that whatever is redistributed isn’t sent back internally via BGP.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
set protocols bgp neighbor 192.168.254.50 address-family ipv4-unicast prefix-list export allow_except_254
set protocols bgp neighbor 192.168.254.51 address-family ipv4-unicast prefix-list export allow_except_254
set protocols bgp neighbor 192.168.254.52 address-family ipv4-unicast prefix-list export allow_except_254
set protocols bgp neighbor 192.168.254.53 address-family ipv4-unicast prefix-list export allow_except_254
set policy route-map redist_254 rule 10 action 'permit'
set policy route-map redist_254 rule 10 match ip address prefix-list 'allow_254_net'
set policy route-map redist_254 rule 20 action 'deny'
set policy prefix-list allow_254_net rule 10 action 'permit'
set policy prefix-list allow_254_net rule 10 le '32'
set policy prefix-list allow_254_net rule 10 prefix '192.168.254.0/24'
set policy prefix-list allow_except_254 rule 10 action 'deny'
set policy prefix-list allow_except_254 rule 10 le '32'
set policy prefix-list allow_except_254 rule 10 prefix '192.168.254.0/24'
set policy prefix-list allow_except_254 rule 20 action 'permit'
set policy prefix-list allow_except_254 rule 20 le '32'
set policy prefix-list allow_except_254 rule 20 prefix '0.0.0.0/0'
set protocols bgp address-family ipv4-unicast redistribute isis route-map 'redist_254'
For the border routers on AS203528 side - handling OSPF <> BGP distribution:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
set policy route-map redist_248 rule 10 action 'permit'
set policy route-map redist_248 rule 10 match ip address prefix-list 'allow_248_net'
set policy route-map redist_248 rule 20 action 'deny'
set policy prefix-list allow_248_net rule 10 action 'permit'
set policy prefix-list allow_248_net rule 10 le '32'
set policy prefix-list allow_248_net rule 10 prefix '192.168.248.0/24'
set policy prefix-list allow_except_248 rule 10 action 'deny'
set policy prefix-list allow_except_248 rule 10 le '32'
set policy prefix-list allow_except_248 rule 10 prefix '192.168.248.0/24'
set policy prefix-list allow_except_248 rule 20 action 'permit'
set policy prefix-list allow_except_248 rule 20 le '32'
set policy prefix-list allow_except_248 rule 20 prefix '0.0.0.0/0'
set policy prefix-list AS203528_v4_OUT rule 20 le 32
set protocols bgp address-family ipv4-unicast redistribute ospf route-map 'redist_248'
set policy route-map redist_254 rule 10 action 'permit'
set policy route-map redist_254 rule 10 match ip address prefix-list 'allow_254_net'
set policy route-map redist_254 rule 20 action 'deny'
set policy prefix-list allow_254_net rule 10 action 'permit'
set policy prefix-list allow_254_net rule 10 le '32'
set policy prefix-list allow_254_net rule 10 prefix '192.168.254.0/24'
set protocols ospf redistribute bgp route-map redist_254
set protocols ospf redistribute bgp metric-type 1
The route reflectors would tag routes coming from each network with a specific community and only advertise back the routes with that (and only that community) to each network - to keep them separate (add path tx all is used). Something like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
set policy community-list OSR_INTERNAL rule 10 regex 65000:444
set policy community-list OSR_INTERNAL rule 10 action permit
set policy community-list AS203528_PUBLIC rule 10 regex 65000:555
set policy community-list AS203528_PUBLIC rule 10 action permit
set policy route-map RTR_OSR1 rule 9 match community community-list AS203528_PUBLIC
set policy route-map RTR_OSR1 rule 9 action deny
set policy route-map RTR_OSR1 rule 10 action 'permit'
set policy route-map RTR_OSR1 rule 10 set community add '65000:101'
set policy route-map RTR_OSR1 rule 10 set community add '65000:444'
set policy route-map RTR_OSR2 rule 9 match community community-list AS203528_PUBLIC
set policy route-map RTR_OSR2 rule 9 action deny
set policy route-map RTR_OSR2 rule 10 action 'permit'
set policy route-map RTR_OSR2 rule 10 set community add '65000:102'
set policy route-map RTR_OSR2 rule 10 set community add '65000:444'
set policy route-map RTR_PUBLIC rule 9 match community community-list OSR_INTERNAL
set policy route-map RTR_PUBLIC rule 9 action deny
set policy route-map RTR_PUBLIC rule 10 action 'permit'
set policy route-map RTR_PUBLIC rule 10 set community add '65000:555'
set policy route-map RTR_OSR_OUTBOUND rule 10 match community community-list AS203528_PUBLIC
set policy route-map RTR_OSR_OUTBOUND rule 10 action deny
set policy route-map RTR_OSR_OUTBOUND rule 20 match community community-list OSR_INTERNAL
set policy route-map RTR_OSR_OUTBOUND rule 20 action permit
set policy route-map RTR_OSR_OUTBOUND rule 30 action deny
set policy route-map RTR_PUBLIC_OUTBOUND rule 10 match community community-list OSR_INTERNAL
set policy route-map RTR_PUBLIC_OUTBOUND rule 10 action deny
set policy route-map RTR_PUBLIC_OUTBOUND rule 20 match community community-list AS203528_PUBLIC
set policy route-map RTR_PUBLIC_OUTBOUND rule 20 action permit
set policy route-map RTR_PUBLIC_OUTBOUND rule 30 action deny
I was then planning to use the private (AS4200000001) reflectors for AS203528 with a simple config:
1
2
3
4
5
6
7
8
9
set protocols bgp neighbor 192.168.248.10 address-family ipv4-unicast addpath-tx-all
set protocols bgp neighbor 192.168.248.10 address-family ipv4-unicast route-reflector-client
set protocols bgp neighbor 192.168.248.10 address-family ipv6-unicast addpath-tx-all
set protocols bgp neighbor 192.168.248.10 address-family ipv6-unicast route-reflector-client
set protocols bgp neighbor 192.168.248.10 description 'AMS1BR1'
set protocols bgp neighbor 192.168.248.10 remote-as 203528
set protocols bgp neighbor 192.168.248.10 local-as 203528
set protocols bgp neighbor 192.168.248.10 timers connect '1'
set protocols bgp neighbor 192.168.248.10 update-source 'dum1'
I found out the hard way that this is not possible. Local & Remote AS cannot be specified to be the same (System AS is AS4200000001)…
Back to planning
The other way I figured out would be adding a VRF to each route reflector. And using something like Juniper’s import-rib. I wasn’t able to figure out how to leak routes from the default instance into the VRF - so I’ve instead created the VRF, with a dummy interface assigned to it. Then two ethernet interfaces connected to each other via the Hypervisor - one on the default instance and other in the VRF. Then the VRF will route through that to literally leak the traffic.
The deployed config on the route reflectors (192.168.254.150 - 192.168.254.153):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
set policy prefix-list allow_vrf_rr rule 10 action 'permit'
set policy prefix-list allow_vrf_rr rule 10 prefix '192.168.254.150/32'
set policy prefix-list allow_vrf_rr rule 11 action 'permit'
set policy prefix-list allow_vrf_rr rule 11 prefix '192.168.254.151/32'
set policy prefix-list allow_vrf_rr rule 12 action 'permit'
set policy prefix-list allow_vrf_rr rule 12 prefix '192.168.254.152/32'
set policy prefix-list allow_vrf_rr rule 13 action 'permit'
set policy prefix-list allow_vrf_rr rule 13 prefix '192.168.254.153/32'
set policy route-map redist_vrf_rr rule 10 action 'permit'
set policy route-map redist_vrf_rr rule 10 match ip address prefix-list 'allow_vrf_rr'
set policy route-map redist_vrf_rr rule 20 action 'deny'
set protocols isis redistribute ipv4 static level-2 route-map redist_vrf_rr
set interfaces dummy dum1 address '192.168.254.153/32'
set interfaces dummy dum1 vrf 'PUBLIC_VRF'
set interfaces dummy dum1 description 'Loopback on Public VRF'
set interfaces ethernet eth2 address '192.168.246.1/30'
set interfaces ethernet eth3 address '192.168.246.2/30'
set interfaces ethernet eth3 vrf 'PUBLIC_VRF'
set interfaces ethernet eth2 description 'Leaking to Public VRF'
set interfaces ethernet eth3 description 'Leaking to Public VRF'
set protocols static route 192.168.254.153/32 interface eth2
set protocols static route 192.168.254.153/32 next-hop 192.168.246.2
set policy route-map DENY-ALL rule 10 action deny
set policy prefix-list6 DENY-ALL-PREF rule 10 action deny
set vrf name PUBLIC_VRF protocols bgp system-as '203528'
set vrf name PUBLIC_VRF protocols bgp parameters route-reflector-allow-outbound-policy
set vrf name PUBLIC_VRF protocols static route 0.0.0.0/0 interface eth3
set vrf name PUBLIC_VRF protocols static route 0.0.0.0/0 next-hop 192.168.246.1
set vrf name PUBLIC_VRF protocols static route6 ::/0 blackhole
set vrf name PUBLIC_VRF table '1000'
set vrf name PUBLIC_VRF ip protocol bgp route-map DENY-ALL
set policy route-map NO_TRANSIT_EXPORT rule 9 action 'deny'
set policy route-map NO_TRANSIT_EXPORT rule 9 match large-community large-community-list 'VIA_TRANSIT'
set policy route-map NO_TRANSIT_EXPORT rule 10 action 'permit'
set policy large-community-list VIA_TRANSIT rule 10 action 'permit'
set policy large-community-list VIA_TRANSIT rule 10 regex '203528:1:1'
eth2 & eth3 are connected to each other. The NO_TRANSIT_EXPORT policy is applied on the RR peerings to transit routers on the cloud to avoid sending other transit prefixes to them (saves ram & costs).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
set protocols bgp neighbor 192.168.248.10 address-family ipv4-unicast addpath-tx-all
set protocols bgp neighbor 192.168.248.10 address-family ipv4-unicast route-reflector-client
set protocols bgp neighbor 192.168.248.10 address-family ipv6-unicast addpath-tx-all
set protocols bgp neighbor 192.168.248.10 address-family ipv6-unicast route-reflector-client
set protocols bgp neighbor 192.168.248.10 address-family ipv6-unicast route-map export NO_TRANSIT_EXPORT
set protocols bgp neighbor 192.168.248.10 description 'AMS1BR1'
set protocols bgp neighbor 192.168.248.10 remote-as internal
set protocols bgp neighbor 192.168.248.10 timers connect '1'
set protocols bgp neighbor 192.168.248.10 update-source 'dum1'
set protocols bgp neighbor 192.168.248.11 address-family ipv4-unicast addpath-tx-all
set protocols bgp neighbor 192.168.248.11 address-family ipv4-unicast route-reflector-client
set protocols bgp neighbor 192.168.248.11 address-family ipv6-unicast addpath-tx-all
set protocols bgp neighbor 192.168.248.11 address-family ipv6-unicast route-reflector-client
set protocols bgp neighbor 192.168.248.11 address-family ipv6-unicast route-map export NO_TRANSIT_EXPORT
set protocols bgp neighbor 192.168.248.11 description 'NYC1BR1'
set protocols bgp neighbor 192.168.248.11 remote-as internal
set protocols bgp neighbor 192.168.248.11 timers connect '1'
set protocols bgp neighbor 192.168.248.11 update-source 'dum1'
set protocols bgp neighbor 192.168.248.12 address-family ipv4-unicast addpath-tx-all
set protocols bgp neighbor 192.168.248.12 address-family ipv4-unicast route-reflector-client
set protocols bgp neighbor 192.168.248.12 address-family ipv6-unicast addpath-tx-all
set protocols bgp neighbor 192.168.248.12 address-family ipv6-unicast route-reflector-client
set protocols bgp neighbor 192.168.248.12 description 'OSR1BR1'
set protocols bgp neighbor 192.168.248.12 remote-as internal
set protocols bgp neighbor 192.168.248.12 timers connect '1'
set protocols bgp neighbor 192.168.248.12 update-source 'dum1'
Results
After deploying the RR configs, the BGP sessions to RRs were deployed on AS203528 and then the iBGP mesh removed.
View from a RR:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
fabrizzio@OSR1RR1:~$ sh bgp vrf PUBLIC_VRF sum
IPv4 Unicast Summary (VRF PUBLIC_VRF):
BGP router identifier 192.168.254.150, local AS number 203528 vrf-id 7
BGP table version 1816
RIB entries 301, using 28 KiB of memory
Peers 13, using 262 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.168.248.10 4 203528 188859 5057 1816 0 0 02:33:06 4 883 AMS1BR1
192.168.248.11 4 203528 71326 3718 1816 0 0 02:11:20 4 883 NYC1BR1
192.168.248.12 4 203528 248 363830 1816 0 0 01:46:03 282 883 OSR1BR1
192.168.248.13 4 203528 2147 367492 1816 0 0 01:59:28 282 883 OSR1BR2
192.168.248.14 4 203528 200 200896 1816 0 0 01:40:43 152 883 OSR2BR1
192.168.248.15 4 203528 171 196763 1816 0 0 01:29:15 152 883 OSR2BR2
192.168.248.16 4 203528 167 531209 1816 0 0 01:08:31 2 883 OSR1BR3
192.168.248.17 4 203528 97 1849 1816 0 0 01:34:46 0 883 OSR2EV1
192.168.248.18 4 203528 63485 3790 1816 0 0 02:10:24 0 883 LUX1BR1
192.168.248.20 4 203528 131 997 1816 0 0 02:07:09 3 883 FFT2EV1
192.168.248.21 4 203528 98 1809 1816 0 0 01:32:43 2 883 FFT2EV2
192.168.248.90 4 203528 98 197908 1816 0 0 01:32:22 0 883 OSR1GLASS1
192.168.248.91 4 203528 102 199461 1816 0 0 01:37:21 0 883 OSR2GLASS1
Total number of neighbors 13
IPv6 Unicast Summary (VRF PUBLIC_VRF):
BGP router identifier 192.168.254.150, local AS number 203528 vrf-id 7
BGP table version 692076
RIB entries 375703, using 34 MiB of memory
Peers 10, using 201 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.168.248.10 4 203528 188859 5057 692076 0 0 02:33:06 200256 3132 AMS1BR1
192.168.248.11 4 203528 71326 3718 692076 0 0 02:11:20 196457 3132 NYC1BR1
192.168.248.12 4 203528 248 363830 692076 0 0 01:46:03 8 594591 OSR1BR1
192.168.248.13 4 203528 2147 367492 692076 0 0 01:59:28 893 594591 OSR1BR2
192.168.248.14 4 203528 200 200896 692076 0 0 01:40:43 22 594591 OSR2BR1
192.168.248.15 4 203528 171 196763 692076 0 0 01:29:15 6 594591 OSR2BR2
192.168.248.16 4 203528 167 531209 692076 0 0 01:08:31 4 594591 OSR1BR3
192.168.248.18 4 203528 63485 3790 692076 0 0 02:10:24 196943 3132 LUX1BR1
192.168.248.90 4 203528 98 197908 692076 0 0 01:32:22 1 594591 OSR1GLASS1
192.168.248.91 4 203528 102 199461 692076 0 0 01:37:21 1 594591 OSR2GLASS1
Total number of neighbors 10
fabrizzio@OSR1RR1:~$
From a router on AS203528:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
fabrizzio@OSR2BR2:~$ sh bgp sum
IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.248.15, local AS number 203528 vrf-id 0
BGP table version 1802
RIB entries 301, using 28 KiB of memory
Peers 5, using 101 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
172.27.2.10 4 4200000001 5689 5682 1802 0 0 01:33:58 130 23 To OSR2FW1
192.168.254.150 4 203528 195409 171 1802 0 0 01:29:45 731 152 To OSR1RR1
192.168.254.151 4 203528 195355 171 1802 0 0 01:29:45 731 152 To OSR1RR2
192.168.254.152 4 203528 195313 171 1802 0 0 01:29:45 731 152 To OSR1RR3
192.168.254.153 4 203528 195357 171 1802 0 0 01:29:45 731 152 To OSR2RR1
Total number of neighbors 5
IPv6 Unicast Summary (VRF default):
BGP router identifier 192.168.248.15, local AS number 203528 vrf-id 0
BGP table version 2330866
RIB entries 375715, using 34 MiB of memory
Peers 5, using 101 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
172.27.2.10 4 4200000001 5689 5682 2330866 0 0 01:33:58 2 5 To OSR2FW1
192.168.254.150 4 203528 195409 171 2330866 0 0 01:29:45 594573 6 To OSR1RR1
192.168.254.151 4 203528 195355 171 2330866 0 0 01:29:45 594573 6 To OSR1RR2
192.168.254.152 4 203528 195313 171 2330866 0 0 01:29:45 594573 6 To OSR1RR3
192.168.254.153 4 203528 195357 171 2330866 0 0 01:29:45 594573 6 To OSR2RR1
Total number of neighbors 5
Route to a RR from AS203528 - proving the IS-IS into BGP -> eBGP -> BGP into OSPF redist works.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
fabrizzio@LUX1BR1:~$ sh ip route 192.168.254.50
Routing entry for 192.168.254.50/32
Known via "bgp", distance 200, metric 100
Last update 01:18:53 ago
192.168.248.12 (recursive), weight 1
172.27.0.166, via wg1, weight 1
192.168.248.13 (recursive), weight 1
172.27.0.197, via wg9, weight 1
192.168.248.15 (recursive), weight 1
172.27.0.170, via wg2, weight 1
Routing entry for 192.168.254.50/32
Known via "ospf", distance 110, metric 2520, best
Last update 01:35:24 ago
* 172.27.0.166, via wg1, weight 1
* 172.27.0.170, via wg2, weight 1
* 172.27.0.197, via wg9, weight 1
The route redistribution is kept under tight control to avoid any routing loops due to the policies deployed on the eBGP peerings.
It all works fine :)
This post is licensed under CC BY 4.0 by the author.