In the IWAN Part 1 PfRv3 Design Considerations blog, I went over several PfRv3 concepts and design considerations. Now that the foundation has been established, we are ready to create a high-level IWAN design. The routing design, however, is the most complex piece of the solution.
PfRv3 is not a routing protocol. It is a piece of software which sits on top of the routing protocol and uses the routing information together with performance measurement statistics in order to make an intelligent forwarding decision.
It not only looks at the routing table, but PfRv3 is also able to analyze the BGP table or the EIGRP topology table. Therefore BGP and EIGRP are the two supported protocols by PfRv3, unlike OSPF and RIP. You may ask which protocol is best for IWAN deployments, EIGRP or BGP. The answer is, it depends as each choice has its pros and cons. The full comparison of the two protocols is out of scope of this article. Nevertheless, if we are talking about a highly scalable solution for thousands of branches then BGP, more precisely iBGP, is considered a better choice.
IWAN-specific routing rules
Once the routing protocol is chosen, we should consider the IWAN-specific routing rules. Some of these rules must be followed to facilitate the behind the scenes PfRv3 functionality, whereas other rules are best practice and optional.
- The Loopback IP address of the Master Controller (MC) at any DC/Branch location should be advertised over each tunnel. The reasons are path discovery, channel creation, probing, and TCAs.
Looking at Picture 1, the arrows are showing the advertisement direction of the Loopback’s IP address which is used by each router for PfRv3 peering. It is mandatory for all Border Routers (BRs) to advertise the Loopback’s IP address of their local MC over the DMVPN tunnels. The Master Controller / Border Router (MC/BR) has two functionalities combined and therefore uses the same Loopback for both of them. In this case, the branches with multiple routers as shown on the bottom right corner of Figure 1 have an interesting situation - although the BR router must advertise the MC/BR’s loopback over its tunnel, the MC/BR router is not required to advertise the BR’s loopback over its tunnel.
- All BR Loopbacks within a given site should be reachable between one another by any means of routing within the site’s local network because an automatic GRE tunnel is used to offload the traffic from one BR to another. If the BR receives a user’s traffic that may need to be offloaded for some reason to an alternative path, the router encapsulates that user’s traffic into the GRE tunnel with the source IP address of its own Loopback and the destination IP address of the Loopback of the alternative BR.
- The MC at the branch should be Layer 3 adjacent with each of the BRs. The reason is that the EIGRP Service-Family (SF) packets TTL used by PfRv3 at the branches is set to 1. The EIGRP Service-Family is needed to exchange the content of site-prefix table.
- The MC running in a given site should be reachable from its local BRs outside of the DMVPN tunnels. This is necessary to establish PfR adjacencies.
- The transport endpoints for the tunnels should never be advertised through the tunnels themselves. The reason is to avoid recursive tunnel issues.
- It is best practice not to advertise the tunnel networks to any routing protocol to avoid routing loops, but ultimately this comes down to a design choice.
- If BGP is used for the overlay network, it is advised that all iBGP peers should be configured with “next-hop-self all”. The reason is to solve issues with unreachable next-hops.
The remainder of the routing design is very flexible and can be chosen depending on the specific requirements.
In this second IWAN blog, I talked about the choice of the supported routing protocols that are mandatory by IWAN, and the best practices routing rules to make the PfRv3 to function correctly. This article builds upon the concepts and design considerations of DMVPN and PfRv3. The third article will talk about crucial effect of microloops on IWAN networks and possible solutions.
About the Author
Dmytro (Dima) Muzychko is a principal design engineer at Verizon and mainly deals with non-standard customer requirements. His experience is built around service provider networking and is strongly focused on routing and switching, virtualization and lastly the SD-WAN. Dima's professional quote is "Intelligence is needed for solving problems; wisdom for avoiding them. Network design is all about the latter."
Dima currently holds a CCDE and a CCIE in Routing & Switching and a Masters in Computer Technologies.
Here are a few additional ways for us to engage and keep the conversation going:
- Cisco Learning Network CCDE Study Group
- Connect on LinkedIn: Dmytro Muzychko
- Connect on Twitter too
- CCDE study materials for the Written and Practical exams
- Related Unleashing CCDE blogs: IWAN Part 1: PfRv3 Design Considerations by Dmytro Muzychko, IWAN Part 3: Microloops on IWAN networks by Dmytro Muzychko, IWAN Part 4: Load Sharing and Load Balancing on IWAN Networks with Dmytro Muzychko, Commercial Solutions for Classified (CSfC) with Joe Galimi, A Network Designer’s Thought Process Part 1 by Cary Chen, Network Function Virtualization in Enterprise by Stephen Lynn, All Smoke and Mirrors by Michael Kowal – Part 1
- Related links: PfRv3:Technology Overview, CVD-IWAN Technology Design Guide Feb-16, CVD-IWAN Configuration Files Guide Feb-16