Hub and Spoke – multiple CE connected to singe routing instance on PE
I was searching for solution quiet a long time but there is no solution / example in training material or internet. So i decide to solve it by myself.
What do we want to achieves?
As we know the main assumption of L3VPN “Hub and Spoke” is, that the traffic between spokes have to pass via hub. This may be because of a firewall in a main branch or other invigilation purpose ;). I will not talk about hub VRF in this post instead of this i will focus only on the spoke end. There is a case when multiple CE are connected to single PE (e.g. two offices in same city).
Of course we can solve this in very simple way, by creating two routing instance / vrf and connect each CE to each instance. But this is extra config and extra routing instance with in some case can be limited by license.
Our goal in this case is to terminate multiple CEs into a singe vrf, and make sure that the traffic between CEs can’t be exchanged via local routing table. It have to pass via hub even it is directed to CE locally connected to same PE.
Protocols
Full hub and spoke scenario will not be covered here. There is a lot of very good blogs, training materials or other documentation covering this in both with dual and single interfaces from central site to PE with hub and spoke vrf.
I’m using here hub and spoke with two interfaces on hub end (one interface with two vlans). BGP is used to for connection between CE and PE. Hub is receiving all prefixes from spokes and it advertise a default to spokes.
Topology
Topology is very simple.
– Central site CE1 connected by separate interfaces to PE on R1 (Hub / Spoke end).
– Spokes – single PE on R3 with two interfaces and CEs terminates in same routing-instance (CE2-spoke)
Prefix distribution
In our scenario EBGP is used between CE2, CE3 and a R3 PE spoke. Same at the hub and spoke end. Prefixes received from CE2 are distributed from R3 spoke routing instance into CE3 and CE1 (as-override is used). We can consider few options here.
We can try to block sending the prefixes from R3 spoke into the CE2 and CE3 by:
– removing as-override
– adding export policy to EBGP
– configuring IBGP
– use static routs etc…
Unfortunately this will not be a solution in this case. No metre with protocol or static will we use forwarding plain always have a next hop pointing to local interface. At the end packet will follow shortest way to the prefix with is local entry in forwarding table (in this case CE2-spoke vrf table on R3). Conclusion is simple we have to get rid of them, and leave only default pointing to hub.
The only solution i can see here is to filter our forwarding plane. After applying filters, we should only have a default 0/0 pointing to hub (CE1) and interface prefixes to both CE2 and CE3. OK, but how will the return traffic sourced from CE2 passing by CE1 hub with a destination of CE3 knows where to forward packets, after reaching our vrf on R3, if we only have an interface routes and 0/0 with direction to CE1?
Since we are not using vrf-table-label (for extra lookup after the mpls header is take off) after striping out VPN label packet is forwarder directly into interface next hop. Here is the explanation founded in juniper training materials:
“The default behaviour of Junos device is to allocate labels on a per next hop basis. A single vrf may have multiple attached CEs. VPN labels are allocated (and advertised by MP-BGP) for each CE-learned VPN route by the PE router based on the PE router’s determination of the best next hop to reach the destination.
When the VPN packet arrivers on agrees PE they are encapsulated in single MPLS label (the VPN label). The forwarding procedure, in the case of per next hop label allocation, is to pop the MPLS label and forward it to the next hop that was learned as described earlier. The IP header is not evaluated for forwarding purposes.”
So the only thing we have to worry about is a local traffic between CE2 and CE3. To limit it we will use the forwarding policy.
Forwarding policy
We are using export policy to allow only 0/0 learned via BGP. We have to apply this on the specific rib group (in this case CE2-spoke.inet.0).
policy-statement def-to-hub-spoke {
term 1 {
from {
route-filter 0.0.0.0/0 exact;
}
to rib CE2-spoke.inet.0;
then accept;
}
term 2 {
to rib CE2-spoke.inet.0;
then reject;
}
}
forwarding-table {
export def-to-hub-spoke;
}
Packet Flow
This is how traffic flows from CE2 to CE3, without any filters applied (directly via R3 spoke rooting instance).
run traceroute source 172.31.77.1 172.31.91.1 routing-instance CE2-5 no-resolve wait 1 traceroute to 172.31.91.1 (172.31.91.1) from 172.31.77.1, 30 hops max, 48 byte packets 1 192.168.0.89 3.821 ms 1.601 ms 2.567 ms 2 172.31.91.1 3.527 ms 2.875 ms 2.214 ms
Lets see how is the forwarding table look like without any modifications
Routing table: CE2-spoke.inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif default user 0 indr 1048575 12 0:5:86:68:b:0 Push 16, Push 300656(top) 591 1 ge-0/0/0.0 default perm 0 rjct 618 1 0.0.0.0/32 perm 0 dscd 615 1 172.30.5.21/32 intf 0 172.30.5.21 locl 641 1 172.30.5.33/32 user 0 indr 1048575 12 172.30.0.81 Push 16, Push 300656(top) 591 1 ge-0/0/0.0 172.30.5.253/32 intf 0 172.30.5.253 locl 642 1 172.31.71.0/24 user 0 indr 1048575 12 172.30.0.81 Push 16, Push 300656(top) 591 1 ge-0/0/0.0 172.31.76.0/24 user 0 192.168.0.90 ucst 592 6 ge-0/0/5.323 172.31.77.0/24 user 0 192.168.0.90 ucst 592 6 ge-0/0/5.323 172.31.91.0/24 user 0 192.168.1.90 ucst 597 5 ge-0/0/5.344 192.168.0.40/30 user 0 indr 1048575 12 172.30.0.81 Push 16, Push 300656(top) 591 1 ge-0/0/0.0 192.168.0.88/30 intf 0 rslv 640 1 ge-0/0/5.323 192.168.0.88/32 dest 0 192.168.0.88 recv 638 1 ge-0/0/5.323 192.168.0.89/32 intf 0 192.168.0.89 locl 639 2 192.168.0.89/32 dest 0 192.168.0.89 locl 639 2 192.168.0.90/32 dest 1 0:5:86:22:e1:1 ucst 592 6 ge-0/0/5.323 192.168.0.91/32 dest 0 192.168.0.91 bcst 637 1 ge-0/0/5.323 192.168.1.88/30 intf 0 rslv 650 1 ge-0/0/5.344 192.168.1.88/32 dest 0 192.168.1.88 recv 648 1 ge-0/0/5.344 192.168.1.89/32 intf 0 192.168.1.89 locl 649 2 192.168.1.89/32 dest 0 192.168.1.89 locl 649 2 192.168.1.90/32 dest 1 0:5:86:22:e1:1 ucst 597 5 ge-0/0/5.344 192.168.1.91/32 dest 0 192.168.1.91 bcst 647 1 ge-0/0/5.344 224.0.0.0/4 perm 0 mdsc 617 1 224.0.0.1/32 perm 0 224.0.0.1 mcst 620 1 255.255.255.255/32 perm 0 bcst 621 1
This is how traffic flows from CE2 to CE3 via CE1, with filters on forwarding plain applied.
traceroute source 172.31.77.1 172.31.91.1 routing-instance CE2 no-resolve wait 1 traceroute to 172.31.91.1 (172.31.91.1) from 172.31.77.1, 30 hops max, 48 byte packets 1 192.168.0.89 10.203 ms 4.787 ms 5.450 ms 2 172.30.0.81 6.586 ms 5.944 ms 4.080 ms MPLS Label=301968 CoS=0 TTL=1 S=0 MPLS Label=16 CoS=0 TTL=1 S=1 3 172.30.5.33 7.622 ms 4.562 ms 4.889 ms 4 192.168.0.42 3.525 ms 4.463 ms 3.889 ms - HUB vrf - CE1 5 192.168.0.45 6.662 ms 3.824 ms 4.18 ms - Spoke vrf - CE1 6 172.30.0.29 8.448 ms 6.107 ms 7.234 ms MPLS Label=301984 CoS=0 TTL=1 S=0 MPLS Label=300112 CoS=0 TTL=1 S=1 7 172.30.0.82 7.215 ms 6.648 ms 8.810 ms MPLS Label=300112 CoS=0 TTL=1 S=1 8 172.31.91.1 11.216 ms 8.072 ms 9.452 ms
This is current routing table for CE2-spoke
show route table CE2-spoke.inet.0 CE2-spoke.inet.0: 13 destinations, 20 routes (13 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 0.0.0.0/0 *[BGP/170] 00:06:04, localpref 100, from 172.30.5.41 AS path: 64600 54591 I, validation-state: unverified > to 172.30.0.81 via ge-0/0/0.0, label-switched-path r4-r7 172.30.5.21/32 *[Direct/0] 2d 05:13:00 > via lo0.1 [BGP/170] 00:06:04, localpref 100, from 172.30.5.41 AS path: 64600 54591 I, validation-state: unverified > to 172.30.0.81 via ge-0/0/0.0, label-switched-path r4-r7 172.30.5.33/32 *[BGP/170] 00:06:04, localpref 100, from 172.30.5.41 AS path: I, validation-state: unverified > to 172.30.0.81 via ge-0/0/0.0, label-switched-path r4-r7 172.30.5.253/32 *[Direct/0] 2d 05:13:00 > via lo0.1 [BGP/170] 00:06:04, localpref 100, from 172.30.5.41 AS path: I, validation-state: unverified > to 172.30.0.81 via ge-0/0/0.0, label-switched-path r3-r1 172.31.71.0/24 *[BGP/170] 00:06:04, localpref 100, from 172.30.5.41 AS path: 64600 I, validation-state: unverified > to 172.30.0.81 via ge-0/0/0.0, label-switched-path r3-r1 172.31.76.0/24 *[BGP/170] 2d 05:12:56, localpref 100 AS path: 64600 I, validation-state: unverified > to 192.168.0.90 via ge-0/0/5.323 [BGP/170] 00:06:04, localpref 100, from 172.30.5.41 AS path: 64600 54591 54591 I, validation-state: unverified > to 172.30.0.81 via ge-0/0/0.0, label-switched-path r3-r1 172.31.77.0/24 *[BGP/170] 2d 05:12:56, localpref 100 AS path: 64600 I, validation-state: unverified > to 192.168.0.90 via ge-0/0/5.323 [BGP/170] 00:06:04, localpref 100, from 172.30.5.41 AS path: 64600 54591 54591 I, validation-state: unverified > to 172.30.0.81 via ge-0/0/0.0, label-switched-path r3-r1 172.31.91.0/24 *[BGP/170] 2d 01:37:34, localpref 100 AS path: 64600 I, validation-state: unverified > to 192.168.1.90 via ge-0/0/5.344 [BGP/170] 00:06:04, localpref 100, from 172.30.5.41 AS path: 64600 54591 54591 I, validation-state: unverified > to 172.30.0.81 via ge-0/0/0.0, label-switched-path r3-r1 192.168.0.40/30 *[BGP/170] 00:06:04, localpref 100, from 172.30.5.41 AS path: I, validation-state: unverified > to 172.30.0.81 via ge-0/0/0.0, label-switched-path r3-r1 192.168.0.88/30 *[Direct/0] 2d 05:13:00 > via ge-0/0/5.323 [BGP/170] 00:06:04, localpref 100, from 172.30.5.41 AS path: 64600 54591 I, validation-state: unverified > to 172.30.0.81 via ge-0/0/0.0, label-switched-path r3-r1 192.168.0.89/32 *[Local/0] 2d 05:13:00 Local via ge-0/0/5.323 192.168.1.88/30 *[Direct/0] 2d 01:38:06 > via ge-0/0/5.344 [BGP/170] 00:06:04, localpref 100, from 172.30.5.41 AS path: 64600 54591 I, validation-state: unverified > to 172.30.0.81 via ge-0/0/0.0, label-switched-path r3-r1 192.168.1.89/32 *[Local/0] 2d 01:38:06 Local via ge-0/0/5.344
Lets see now how does the forwarding looks like with filter applied.
run show route forwarding-table table CE2-spoke Routing table: CE2-spoke.inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif default user 0 indr 1048574 2 0:5:86:68:b:0 Push 16, Push 301968(top) 591 2 ge-0/0/0.0 default perm 0 rjct 618 1 0.0.0.0/32 perm 0 dscd 615 1 172.30.5.21/32 intf 0 172.30.5.21 locl 641 1 172.30.5.253/32 intf 0 172.30.5.253 locl 642 1 192.168.0.88/30 intf 0 rslv 640 1 ge-0/0/5.323 192.168.0.88/32 dest 0 192.168.0.88 recv 638 1 ge-0/0/5.323 192.168.0.89/32 intf 0 192.168.0.89 locl 639 2 192.168.0.89/32 dest 0 192.168.0.89 locl 639 2 192.168.0.90/32 dest 1 0:5:86:22:e1:1 ucst 592 4 ge-0/0/5.323 192.168.0.91/32 dest 0 192.168.0.91 bcst 637 1 ge-0/0/5.323 192.168.1.88/30 intf 0 rslv 650 1 ge-0/0/5.344 192.168.1.88/32 dest 0 192.168.1.88 recv 648 1 ge-0/0/5.344 192.168.1.89/32 intf 0 192.168.1.89 locl 649 2 192.168.1.89/32 dest 0 192.168.1.89 locl 649 2 192.168.1.90/32 dest 1 0:5:86:22:e1:1 ucst 597 4 ge-0/0/5.344 192.168.1.91/32 dest 0 192.168.1.91 bcst 647 1 ge-0/0/5.344 224.0.0.0/4 perm 0 mdsc 617 1 224.0.0.1/32 perm 0 224.0.0.1 mcst 620 1 255.255.255.255/32 perm 0 bcst 621 1
Routers configuration – do it yourself
Router R3 – routing instance and policy.
show routing-instances
CE2-spoke {
instance-type vrf;
interface ge-0/0/5.323;
interface ge-0/0/5.344;
interface lo0.1;
vrf-import CE2-spoke-import;
vrf-export CE2-spoke-export;
protocols {
bgp {
group ce {
type external;
peer-as 64600;
as-override;
neighbor 192.168.0.90;
neighbor 192.168.1.90;
}
}
}
}
show policy-options
policy-statement CE2-spoke-export {
term 1 {
from protocol [ direct bgp ];
then {
community add CE2-spoke;
accept;
}
}
}
policy-statement CE2-spoke-import {
term 2 {
from {
protocol bgp;
community CE2-hub;
}
then accept;
}
}
policy-statement def-to-hub-spoke {
term 1 {
from {
route-filter 0.0.0.0/0 exact;
}
to rib CE2-spoke.inet.0;
then accept;
}
term 2 {
to rib CE2-spoke.inet.0;
then reject;
}
}
}
community CE2-hub members target:54591:200;
community CE2-spoke members target:54591:201;
routing-options {
route-distinguisher-id 172.30.5.4;
autonomous-system 54591 loops 3;
forwarding-table {
export def-to-hub-spoke;
}
}
Router R1 – Hub and Spoke
routing-instances {
CE2-hub {
instance-type vrf;
interface ge-0/0/5.311;
interface lo0.1;
vrf-import CE2-hub-import;
vrf-export CE2-hub-export;
vrf-table-label;
protocols {
bgp {
group ce {
type external;
peer-as 64600;
as-override;
neighbor 192.168.0.42;
}
}
}
}
CE2-spoke {
instance-type vrf;
interface ge-0/0/5.312;
interface lo0.2;
vrf-import CE2-spoke-import;
vrf-export CE2-spoke-export;
vrf-table-label;
routing-options {
static {
route 0.0.0.0/0 next-table inet.0;
}
}
protocols {
bgp {
group ce {
type external;
export default-to-ce;
peer-as 64600;
as-override;
neighbor 192.168.0.46;
}
}
}
}
}
policy-options {
policy-statement CE1-import {
term 1 {
from {
protocol bgp;
community CE1;
}
then accept;
}
}
policy-statement CE2-hub-export {
term 1 {
from protocol [ direct bgp ];
then {
community add CE2-hub;
accept;
}
}
}
policy-statement CE2-hub-import {
term 1 {
then reject;
}
}
policy-statement CE2-spoke-export {
term 1 {
then reject;
}
}
policy-statement CE2-spoke-import {
term 1 {
from {
protocol bgp;
community CE2-spoke;
}
then accept;
}
}
policy-statement default-to-ce {
term 1 {
from {
protocol static;
route-filter 0.0.0.0/0 exact;
}
then accept;
}
}
community CE2-hub members target:54591:200;
community CE2-spoke members target:54591:201;
}
routing-options {
route-distinguisher-id 172.30.5.7;
autonomous-system 54591 loops 3;
}
From the author 😉
I hope this will be useful to somebody.