My hobby network (AS203528) keeps growing. I had a full iBGP mesh among those routers, however that's no longer viable.

I already have 4xBGP Route reflectors on my internal network. I want to re-use those same reflectors for AS203528. Those two networks have each their own IGP, and are just peered with each other.

The Plan

My entire network needs to be able to pass a cold start / bootstrap itself. I have the loopbacks of my internal network on 192.168.254.0/24 and of AS203528 at 192.168.248.0/24. Even though each loopback subnet is a /32, only the /24 crosses network boundaries.

I don't think that it's a good idea for the connectivity from AS203528 to/from its route reflectors to depend on this summarized prefix. Instead I want the /32 loopbacks to cross networks, to prevent a network split or router isolation to cause major impact.

Obviously the main point is that AS203528 currently knows 192.168.254.0/24 via BGP. The AS203528 border routers that peer with the private network know it via eBGP and the rest of AS203528 via iBGP. This can't be the case anymore, otherwise how can the network bootstrap itself if it needs BGP to reach its route reflectors?.

Therefore I am going to distribute the individual /32s from 192.168.254.0/24 from IS-IS into BGP on my private network - then send them to AS203528 via eBGP, and then redistribute them from BGP into OSPF within AS203528. This way the bootstrap will work.

I apologize for the crappy diagrams.

Overview

This diagram will explain how the route reflector loopbacks will flow from private into AS203528:

Private -> AS203528

And how the loopbacks from AS203528 flow into the private network:

AS203528 -> Private

Steps

First I started by redistributing the required prefixes from IS-IS into BGP at the Firewall/Routers on the Private side. I just make sure that whatever is redistributed isn't sent back internally via BGP.

 1set protocols bgp neighbor 192.168.254.50 address-family ipv4-unicast prefix-list export allow_except_254
 2set protocols bgp neighbor 192.168.254.51 address-family ipv4-unicast prefix-list export allow_except_254
 3set protocols bgp neighbor 192.168.254.52 address-family ipv4-unicast prefix-list export allow_except_254
 4set protocols bgp neighbor 192.168.254.53 address-family ipv4-unicast prefix-list export allow_except_254
 5set policy route-map redist_254 rule 10 action 'permit'
 6set policy route-map redist_254 rule 10 match ip address prefix-list 'allow_254_net'
 7set policy route-map redist_254 rule 20 action 'deny'
 8set policy prefix-list allow_254_net rule 10 action 'permit'
 9set policy prefix-list allow_254_net rule 10 le '32'
10set policy prefix-list allow_254_net rule 10 prefix '192.168.254.0/24'
11set policy prefix-list allow_except_254 rule 10 action 'deny'
12set policy prefix-list allow_except_254 rule 10 le '32'
13set policy prefix-list allow_except_254 rule 10 prefix '192.168.254.0/24'
14set policy prefix-list allow_except_254 rule 20 action 'permit'
15set policy prefix-list allow_except_254 rule 20 le '32'
16set policy prefix-list allow_except_254 rule 20 prefix '0.0.0.0/0'
17set protocols bgp address-family ipv4-unicast redistribute isis route-map 'redist_254'

For the border routers on AS203528 side - handling OSPF <> BGP distribution:

 1set policy route-map redist_248 rule 10 action 'permit'
 2set policy route-map redist_248 rule 10 match ip address prefix-list 'allow_248_net'
 3set policy route-map redist_248 rule 20 action 'deny'
 4set policy prefix-list allow_248_net rule 10 action 'permit'
 5set policy prefix-list allow_248_net rule 10 le '32'
 6set policy prefix-list allow_248_net rule 10 prefix '192.168.248.0/24'
 7set policy prefix-list allow_except_248 rule 10 action 'deny'
 8set policy prefix-list allow_except_248 rule 10 le '32'
 9set policy prefix-list allow_except_248 rule 10 prefix '192.168.248.0/24'
10set policy prefix-list allow_except_248 rule 20 action 'permit'
11set policy prefix-list allow_except_248 rule 20 le '32'
12set policy prefix-list allow_except_248 rule 20 prefix '0.0.0.0/0'
13set policy prefix-list AS203528_v4_OUT rule 20 le 32
14set protocols bgp address-family ipv4-unicast redistribute ospf route-map 'redist_248'
15
16set policy route-map redist_254 rule 10 action 'permit'
17set policy route-map redist_254 rule 10 match ip address prefix-list 'allow_254_net'
18set policy route-map redist_254 rule 20 action 'deny'
19set policy prefix-list allow_254_net rule 10 action 'permit'
20set policy prefix-list allow_254_net rule 10 le '32'
21set policy prefix-list allow_254_net rule 10 prefix '192.168.254.0/24'
22set protocols ospf redistribute bgp route-map redist_254
23set protocols ospf redistribute bgp metric-type 1

The route reflectors would tag routes coming from each network with a specific community and only advertise back the routes with that (and only that community) to each network - to keep them separate (add path tx all is used). Something like this:

 1set policy community-list OSR_INTERNAL rule 10 regex 65000:444
 2set policy community-list OSR_INTERNAL rule 10 action permit
 3
 4set policy community-list AS203528_PUBLIC rule 10 regex 65000:555
 5set policy community-list AS203528_PUBLIC rule 10 action permit
 6
 7set policy route-map RTR_OSR1 rule 9 match community community-list AS203528_PUBLIC
 8set policy route-map RTR_OSR1 rule 9 action deny
 9set policy route-map RTR_OSR1 rule 10 action 'permit'
10set policy route-map RTR_OSR1 rule 10 set community add '65000:101'
11set policy route-map RTR_OSR1 rule 10 set community add '65000:444'
12
13set policy route-map RTR_OSR2 rule 9 match community community-list AS203528_PUBLIC
14set policy route-map RTR_OSR2 rule 9 action deny
15set policy route-map RTR_OSR2 rule 10 action 'permit'
16set policy route-map RTR_OSR2 rule 10 set community add '65000:102'
17set policy route-map RTR_OSR2 rule 10 set community add '65000:444'
18
19set policy route-map RTR_PUBLIC rule 9 match community community-list OSR_INTERNAL
20set policy route-map RTR_PUBLIC rule 9 action deny
21set policy route-map RTR_PUBLIC rule 10 action 'permit'
22set policy route-map RTR_PUBLIC rule 10 set community add '65000:555'
23
24set policy route-map RTR_OSR_OUTBOUND rule 10 match community community-list AS203528_PUBLIC
25set policy route-map RTR_OSR_OUTBOUND rule 10 action deny
26set policy route-map RTR_OSR_OUTBOUND rule 20 match community community-list OSR_INTERNAL
27set policy route-map RTR_OSR_OUTBOUND rule 20 action permit
28set policy route-map RTR_OSR_OUTBOUND rule 30 action deny
29
30set policy route-map RTR_PUBLIC_OUTBOUND rule 10 match community community-list OSR_INTERNAL
31set policy route-map RTR_PUBLIC_OUTBOUND rule 10 action deny
32set policy route-map RTR_PUBLIC_OUTBOUND rule 20 match community community-list AS203528_PUBLIC
33set policy route-map RTR_PUBLIC_OUTBOUND rule 20 action permit
34set policy route-map RTR_PUBLIC_OUTBOUND rule 30 action deny

I was then planning to use the private (AS4200000001) reflectors for AS203528 with a simple config:

1set protocols bgp neighbor 192.168.248.10 address-family ipv4-unicast addpath-tx-all
2set protocols bgp neighbor 192.168.248.10 address-family ipv4-unicast route-reflector-client
3set protocols bgp neighbor 192.168.248.10 address-family ipv6-unicast addpath-tx-all
4set protocols bgp neighbor 192.168.248.10 address-family ipv6-unicast route-reflector-client
5set protocols bgp neighbor 192.168.248.10 description 'AMS1BR1'
6set protocols bgp neighbor 192.168.248.10 remote-as 203528
7set protocols bgp neighbor 192.168.248.10 local-as 203528
8set protocols bgp neighbor 192.168.248.10 timers connect '1'
9set protocols bgp neighbor 192.168.248.10 update-source 'dum1'

I found out the hard way that this is not possible. Local & Remote AS cannot be specified to be the same (System AS is AS4200000001)...

Back to planning

The other way I figured out would be adding a VRF to each route reflector. And using something like Juniper's import-rib. I wasn't able to figure out how to leak routes from the default instance into the VRF - so I've instead created the VRF, with a dummy interface assigned to it. Then two ethernet interfaces connected to each other via the Hypervisor - one on the default instance and other in the VRF. Then the VRF will route through that to literally leak the traffic.

The deployed config on the route reflectors (192.168.254.150 - 192.168.254.153):

 1set policy prefix-list allow_vrf_rr rule 10 action 'permit'
 2set policy prefix-list allow_vrf_rr rule 10 prefix '192.168.254.150/32'
 3set policy prefix-list allow_vrf_rr rule 11 action 'permit'
 4set policy prefix-list allow_vrf_rr rule 11 prefix '192.168.254.151/32'
 5set policy prefix-list allow_vrf_rr rule 12 action 'permit'
 6set policy prefix-list allow_vrf_rr rule 12 prefix '192.168.254.152/32'
 7set policy prefix-list allow_vrf_rr rule 13 action 'permit'
 8set policy prefix-list allow_vrf_rr rule 13 prefix '192.168.254.153/32'
 9
10set policy route-map redist_vrf_rr rule 10 action 'permit'
11set policy route-map redist_vrf_rr rule 10 match ip address prefix-list 'allow_vrf_rr'
12set policy route-map redist_vrf_rr rule 20 action 'deny'
13
14set protocols isis redistribute ipv4 static level-2 route-map redist_vrf_rr
15
16set interfaces dummy dum1 address '192.168.254.153/32'
17set interfaces dummy dum1 vrf 'PUBLIC_VRF'
18set interfaces dummy dum1 description 'Loopback on Public VRF'
19set interfaces ethernet eth2 address '192.168.246.1/30'
20set interfaces ethernet eth3 address '192.168.246.2/30'
21set interfaces ethernet eth3 vrf 'PUBLIC_VRF'
22set interfaces ethernet eth2 description 'Leaking to Public VRF'
23set interfaces ethernet eth3 description 'Leaking to Public VRF'
24
25set protocols static route 192.168.254.153/32 interface eth2
26set protocols static route 192.168.254.153/32 next-hop 192.168.246.2
27
28set policy route-map DENY-ALL rule 10 action deny
29set policy prefix-list6 DENY-ALL-PREF rule 10 action deny
30
31set vrf name PUBLIC_VRF protocols bgp system-as '203528'
32set vrf name PUBLIC_VRF protocols bgp parameters route-reflector-allow-outbound-policy
33set vrf name PUBLIC_VRF protocols static route 0.0.0.0/0 interface eth3
34set vrf name PUBLIC_VRF protocols static route 0.0.0.0/0 next-hop 192.168.246.1
35set vrf name PUBLIC_VRF protocols static route6 ::/0 blackhole
36set vrf name PUBLIC_VRF table '1000'
37set vrf name PUBLIC_VRF ip protocol bgp route-map DENY-ALL
38
39set policy route-map NO_TRANSIT_EXPORT rule 9 action 'deny'
40set policy route-map NO_TRANSIT_EXPORT rule 9 match large-community large-community-list 'VIA_TRANSIT'
41set policy route-map NO_TRANSIT_EXPORT rule 10 action 'permit'
42set policy large-community-list VIA_TRANSIT rule 10 action 'permit'
43set policy large-community-list VIA_TRANSIT rule 10 regex '203528:1:1'

eth2 & eth3 are connected to each other. The NO_TRANSIT_EXPORT policy is applied on the RR peerings to transit routers on the cloud to avoid sending other transit prefixes to them (saves ram & costs).

 1set protocols bgp neighbor 192.168.248.10 address-family ipv4-unicast addpath-tx-all
 2set protocols bgp neighbor 192.168.248.10 address-family ipv4-unicast route-reflector-client
 3set protocols bgp neighbor 192.168.248.10 address-family ipv6-unicast addpath-tx-all
 4set protocols bgp neighbor 192.168.248.10 address-family ipv6-unicast route-reflector-client
 5set protocols bgp neighbor 192.168.248.10 address-family ipv6-unicast route-map export NO_TRANSIT_EXPORT
 6set protocols bgp neighbor 192.168.248.10 description 'AMS1BR1'
 7set protocols bgp neighbor 192.168.248.10 remote-as internal
 8set protocols bgp neighbor 192.168.248.10 timers connect '1'
 9set protocols bgp neighbor 192.168.248.10 update-source 'dum1'
10
11
12set protocols bgp neighbor 192.168.248.11 address-family ipv4-unicast addpath-tx-all
13set protocols bgp neighbor 192.168.248.11 address-family ipv4-unicast route-reflector-client
14set protocols bgp neighbor 192.168.248.11 address-family ipv6-unicast addpath-tx-all
15set protocols bgp neighbor 192.168.248.11 address-family ipv6-unicast route-reflector-client
16set protocols bgp neighbor 192.168.248.11 address-family ipv6-unicast route-map export NO_TRANSIT_EXPORT
17set protocols bgp neighbor 192.168.248.11 description 'NYC1BR1'
18set protocols bgp neighbor 192.168.248.11 remote-as internal
19set protocols bgp neighbor 192.168.248.11 timers connect '1'
20set protocols bgp neighbor 192.168.248.11 update-source 'dum1'
21
22
23set protocols bgp neighbor 192.168.248.12 address-family ipv4-unicast addpath-tx-all
24set protocols bgp neighbor 192.168.248.12 address-family ipv4-unicast route-reflector-client
25set protocols bgp neighbor 192.168.248.12 address-family ipv6-unicast addpath-tx-all
26set protocols bgp neighbor 192.168.248.12 address-family ipv6-unicast route-reflector-client
27set protocols bgp neighbor 192.168.248.12 description 'OSR1BR1'
28set protocols bgp neighbor 192.168.248.12 remote-as internal
29set protocols bgp neighbor 192.168.248.12 timers connect '1'
30set protocols bgp neighbor 192.168.248.12 update-source 'dum1'

Results

After deploying the RR configs, the BGP sessions to RRs were deployed on AS203528 and then the iBGP mesh removed.

View from a RR:

 1fabrizzio@OSR1RR1:~$ sh bgp vrf PUBLIC_VRF sum
 2
 3IPv4 Unicast Summary (VRF PUBLIC_VRF):
 4BGP router identifier 192.168.254.150, local AS number 203528 vrf-id 7
 5BGP table version 1816
 6RIB entries 301, using 28 KiB of memory
 7Peers 13, using 262 KiB of memory
 8
 9Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
10192.168.248.10  4     203528    188859      5057     1816    0    0 02:33:06            4      883 AMS1BR1
11192.168.248.11  4     203528     71326      3718     1816    0    0 02:11:20            4      883 NYC1BR1
12192.168.248.12  4     203528       248    363830     1816    0    0 01:46:03          282      883 OSR1BR1
13192.168.248.13  4     203528      2147    367492     1816    0    0 01:59:28          282      883 OSR1BR2
14192.168.248.14  4     203528       200    200896     1816    0    0 01:40:43          152      883 OSR2BR1
15192.168.248.15  4     203528       171    196763     1816    0    0 01:29:15          152      883 OSR2BR2
16192.168.248.16  4     203528       167    531209     1816    0    0 01:08:31            2      883 OSR1BR3
17192.168.248.17  4     203528        97      1849     1816    0    0 01:34:46            0      883 OSR2EV1
18192.168.248.18  4     203528     63485      3790     1816    0    0 02:10:24            0      883 LUX1BR1
19192.168.248.20  4     203528       131       997     1816    0    0 02:07:09            3      883 FFT2EV1
20192.168.248.21  4     203528        98      1809     1816    0    0 01:32:43            2      883 FFT2EV2
21192.168.248.90  4     203528        98    197908     1816    0    0 01:32:22            0      883 OSR1GLASS1
22192.168.248.91  4     203528       102    199461     1816    0    0 01:37:21            0      883 OSR2GLASS1
23
24Total number of neighbors 13
25
26IPv6 Unicast Summary (VRF PUBLIC_VRF):
27BGP router identifier 192.168.254.150, local AS number 203528 vrf-id 7
28BGP table version 692076
29RIB entries 375703, using 34 MiB of memory
30Peers 10, using 201 KiB of memory
31
32Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
33192.168.248.10  4     203528    188859      5057   692076    0    0 02:33:06       200256     3132 AMS1BR1
34192.168.248.11  4     203528     71326      3718   692076    0    0 02:11:20       196457     3132 NYC1BR1
35192.168.248.12  4     203528       248    363830   692076    0    0 01:46:03            8   594591 OSR1BR1
36192.168.248.13  4     203528      2147    367492   692076    0    0 01:59:28          893   594591 OSR1BR2
37192.168.248.14  4     203528       200    200896   692076    0    0 01:40:43           22   594591 OSR2BR1
38192.168.248.15  4     203528       171    196763   692076    0    0 01:29:15            6   594591 OSR2BR2
39192.168.248.16  4     203528       167    531209   692076    0    0 01:08:31            4   594591 OSR1BR3
40192.168.248.18  4     203528     63485      3790   692076    0    0 02:10:24       196943     3132 LUX1BR1
41192.168.248.90  4     203528        98    197908   692076    0    0 01:32:22            1   594591 OSR1GLASS1
42192.168.248.91  4     203528       102    199461   692076    0    0 01:37:21            1   594591 OSR2GLASS1
43
44Total number of neighbors 10
45fabrizzio@OSR1RR1:~$

From a router on AS203528:

 1fabrizzio@OSR2BR2:~$ sh bgp sum
 2
 3IPv4 Unicast Summary (VRF default):
 4BGP router identifier 192.168.248.15, local AS number 203528 vrf-id 0
 5BGP table version 1802
 6RIB entries 301, using 28 KiB of memory
 7Peers 5, using 101 KiB of memory
 8
 9Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
10172.27.2.10     4 4200000001      5689      5682     1802    0    0 01:33:58          130       23 To OSR2FW1
11192.168.254.150 4     203528    195409       171     1802    0    0 01:29:45          731      152 To OSR1RR1
12192.168.254.151 4     203528    195355       171     1802    0    0 01:29:45          731      152 To OSR1RR2
13192.168.254.152 4     203528    195313       171     1802    0    0 01:29:45          731      152 To OSR1RR3
14192.168.254.153 4     203528    195357       171     1802    0    0 01:29:45          731      152 To OSR2RR1
15
16Total number of neighbors 5
17
18IPv6 Unicast Summary (VRF default):
19BGP router identifier 192.168.248.15, local AS number 203528 vrf-id 0
20BGP table version 2330866
21RIB entries 375715, using 34 MiB of memory
22Peers 5, using 101 KiB of memory
23
24Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
25172.27.2.10     4 4200000001      5689      5682  2330866    0    0 01:33:58            2        5 To OSR2FW1
26192.168.254.150 4     203528    195409       171  2330866    0    0 01:29:45       594573        6 To OSR1RR1
27192.168.254.151 4     203528    195355       171  2330866    0    0 01:29:45       594573        6 To OSR1RR2
28192.168.254.152 4     203528    195313       171  2330866    0    0 01:29:45       594573        6 To OSR1RR3
29192.168.254.153 4     203528    195357       171  2330866    0    0 01:29:45       594573        6 To OSR2RR1
30
31Total number of neighbors 5

Route to a RR from AS203528 - proving the IS-IS into BGP -> eBGP -> BGP into OSPF redist works.

 1fabrizzio@LUX1BR1:~$ sh ip route 192.168.254.50
 2Routing entry for 192.168.254.50/32
 3  Known via "bgp", distance 200, metric 100
 4  Last update 01:18:53 ago
 5    192.168.248.12 (recursive), weight 1
 6      172.27.0.166, via wg1, weight 1
 7    192.168.248.13 (recursive), weight 1
 8      172.27.0.197, via wg9, weight 1
 9    192.168.248.15 (recursive), weight 1
10      172.27.0.170, via wg2, weight 1
11
12Routing entry for 192.168.254.50/32
13  Known via "ospf", distance 110, metric 2520, best
14  Last update 01:35:24 ago
15  * 172.27.0.166, via wg1, weight 1
16  * 172.27.0.170, via wg2, weight 1
17  * 172.27.0.197, via wg9, weight 1

The route redistribution is kept under tight control to avoid any routing loops due to the policies deployed on the eBGP peerings.

It all works fine :)