In this article, I am going to describe a very important aspect of IWAN that network designers should be very careful about, the PfRv3 auto-discovery of the local prefixes at the branches. An incorrect decision may lead to eventual outage of the whole network.
PfRv3 HUBs and Transit sites have to be manually set with the prefix-list containing the networks hosted at the site, however for the branches it is not mandatory and you may leave branch routers to discover the local networks automatically. If a branch Master Controller (MC) is configured to rely on auto-discovery of the local prefixes, which is the default state, it will wait for its Border Routers (BRs) to report the discovered prefixes. The BRs have a so called “Performance Monitor” (PM) assigned to its tunnel interface in the outbound direction. This PM is checking the source IP address of each packet forwarded from the LAN to DMVPN Tunnel, and then it pulls from the routing table the prefix mask which covers the newly discovered IP address and reports it as a local network belonging to this branch.
This automatic local network discovery sounds good in theory, however Cisco recommends using manual PfRv3 site-prefix configuration at the branches since it could cause very tricky network failures. There could suddenly be some data center (DC) prefixes that start to become advertised within PfRv3 from a particular branch site. Then the traffic destined to the impacted prefixes from the whole network will be sent to that branch over DMVPN and eventually the router will go beyond its supported performance capacity.
At some point in time there is a packet coming to the router from the LAN with a source IP address belonging to the DC. That causes the PM to discover it and claim it as a local subnet. So, where is the problem? The first thing which comes to mind is that the cause of the issue is a routing loop, but checking the routing will prove that all is good. Yet for some reason the DC (HUB) prefixes are advertised in PfRv3 by the MC from a branch site.
Here is what actually happens. Let’s imagine an average DMVPN network, the DMVPN could be either Phase 1, Phase 2 or Phase 3. Let’s say we use BGP (iBGP) as an overlay protocol running inside DMVPN between HUBs (DCs) and branches. The HUB BRs advertise to the branches the summary addresses covering all branches and DC networks, or even a default route. In some cases the HUB should act as either a primary or a backup Direct Internet Access (DIA). In addition, the DC BRs are also advertising their local more specific networks, as with multiple HUBs advertising only summary routes you would have suboptimal routing for the traffic destined to the HUBs themselves. Now, the branch site is redistributing the LAN subnets from its IGP to BGP which then advertises them out to the DC. Please refer to the Figure 1.
During normal operations, there is always traffic between the DC and the branch. Now, let’s imagine that one of the LAN subnets suddenly went down due to an internal switch reloading. The LAN devices will detect the failure and start to withdraw the impacted LAN subnet from the IGP database. The packets will still be sent to the LAN until the BRs are converged. Please refer to Figure 2.
At this step, the BRs are sending the packets to the router LAN R1, but as LAN R1 is already converged it sends the packets back to BR due to the summary (or default route) received from DC. The packets will bounce back and forth until their TTL expired. During this packet bouncing the LAN R1 router eventually updates the BRs with the network change in the LAN and the BRs start withdrawing the unreachable network from their routing tables. If at this exact moment when the BRs are converged they receive the bouncing packet from the LAN, they send it to the DC due to the summary route. It could even be a single packet, but what is the source IP of it? It is an IP address from the DC! At this step, PM discovers the source IP and claims that this subnet is hosted locally at the branch.
The root cause of the problem has a name – “microloop”, and it normally takes place between two devices which have inconsistent routing information during the convergence process. Even though these microloops only lasts an extremely short amount of time, it is more than enough time to mess up the whole IWAN network with auto-discovery turned on at the branches.
To sum it up, microloops occur whenever and wherever the convergence process takes place, but they have a big impact on IWAN networks, when DC prefixes are advertised from a branch and eventually becomes unreachable. Microloops are just the root cause, but when the following conditions are met the whole IWAN network is impacted:
- The branch sites are left in auto-discovery mode and are running some IGP
- The HUB routers advertise a summary or default route comprising the branch networks as well
- There is real traffic coming from DC to branch hosts
- A network failure occur on the branch LAN
The first three items are very common. The last item normally comes at the worst time for an engineer – during the weekend, because of a planned maintenance happening at the branch.
So, how to prevent an IWAN network outage caused by microloops?
- Perform route summarization at the branch sites as much as possible.
- Disable the PfRv3 site prefix-list auto-discovery by manual configuration of the PfRv3 site prefix-list (Cisco best practices for IWAN networks).
About the Author
Dmytro (Dima) Muzychko is a principal design engineer at Verizon and mainly deals with non-standard customer requirements. His experience is built around service provider networking and is strongly focused on routing and switching, virtualization and lastly the SD-WAN. Dima's professional quote is "Intelligence is needed for solving problems; wisdom for avoiding them. Network design is all about the latter."
Dima currently holds a CCDE and a CCIE in Routing & Switching and a Masters in Computer Technologies.
Here are a few additional ways for us to engage and keep the conversation going:
- Cisco Learning Network CCDE Study Group
- Connect on LinkedIn: Dmytro Muzychko
- Connect on Twitter too
- CCDE study materials for the Written and Practical exams
- Related Unleashing CCDE blogs: IWAN Part 1: PfRv3 Design Considerations by Dmytro Muzychko, IWAN Part 2: Routing Rules by Dmytro Muzychko, IWAN Part 4: Load Sharing and Load Balancing on IWAN Networks with Dmytro Muzychko, Commercial Solutions for Classified (CSfC) with Joe Galimi, A Network Designer’s Thought Process Part 1 by Cary Chen, Network Function Virtualization in Enterprise by Stephen Lynn, All Smoke and Mirrors by Michael Kowal – Part 1