1 2 Previous Next 21 Replies Latest reply: Jul 20, 2019 7:55 AM by Sakar RSS

    Default route ECMP Load Balancing abnormal routing

    Sakar

      How does ECMP Load Balancing algorithm work for incoming traffic?

       

      I tried to simulate the scenario in GNS3 and have got varying results.

      Here is my topology.

       

      Topology.png

       

      R1 is customer router, R2 is ISP1, R3 is ISP2, PC-2 and PC-3 are hosts on the internet.

      Assume 1.1.1.0 /24 and 4.4.4.0 /24 blocks owned by R2 (ISP1). 2.2.2.0 /24 and 5.5.5.0 /24 blocks owned by R3 (ISP2). 3.3.3.0 /24 block can be assumed to be owned by either ISP.


      R1 has two default static routes; one to ISP1 another to ISP2.

       

      ISPs will generally employ BCP38 (filtering source address, allowing source IPs within their network at the edge router). Here, I have created standard ACL on R2 allowing only 1.1.1.0 /24 network source IPs to ingress through f0/0 interface (1.1.1.1). Similar configuration on R3 allowing only 2.2.2.0 /24 network source IPs to ingress through f0/0 interface (2.2.2.1).

       

      Because of BCP38 R1 should forward return traffic of incoming traffic from internet through the same interface in which it was received. If it chooses interface connected to another ISP (maybe due to LB logic), the other ISP will drop the traffic.

       

      This is what I expected. But for multiple simulation runs I have obtained different results.

       

      Case I: R1 routes return traffic from the same interface on which request traffic was received (as per my expectation) making pings from PC-2 and PC-3 to R1's public IP 1.1.1.2 and 2.2.2.2 successful.

       

      Case 2: Pings from PC-2 to R1's public IP 1.1.1.2 and 2.2.2.2 are successful but pings from PC-3 to R1's public IP 1.1.1.2 and 2.2.2.2 are unsuccessful. Pings for 1.1.1.2 are unsuccessful because Ping reply for IP 1.1.1.2 is forwarded to R3 (2.2.2.1) which is dropped by R3. Similarly, Pings for 2.2.2.2 are unsuccessful because Ping reply for IP 2.2.2.2 is forwarded to R2 (1.1.1.1) which is also dropped.

       

      Case 3: Vice-Versa case of case 2 pings from PC-3 are successful while pings from PC-2 are unsuccessful.

       

      Case 4: Pings from both PC-2 and PC-3 fail.

       

      Here are screenshots for Case 3.

       

      PC2 ping R1; PC2 console.png

      Pings from PC-2 fail

       

      PC2 ping R1; R1 console.png

      Debug messages on R1 when pings from PC-2 failed

       

      PC3 ping R1; PC3 console.png

      Pings from PC-3 are successful

       

      PC3 ping R1; R1 console.png

      Debug messages on R1 when pings from PC-3 are successful

       

      R1 routes.png

      Default Routes on R1

       

      R1 Configuration

      =================================================================================

      hostname R1

      !

      interface FastEthernet0/0

      ip address 1.1.1.2 255.255.255.0

      ip nat outside

      ip virtual-reassembly in

      duplex auto

      speed auto

      !

      interface FastEthernet0/1

      ip address 2.2.2.2 255.255.255.0

      ip nat outside

      ip virtual-reassembly in

      duplex auto

      speed auto

      !

      interface FastEthernet1/0

      ip address 192.168.1.1 255.255.255.0

      ip nat inside

      ip virtual-reassembly in

      duplex auto

      speed auto

      !

      interface FastEthernet1/1

      no ip address

      shutdown

      duplex auto

      speed auto

      !

      ip nat inside source route-map NAT_ISP1 interface FastEthernet0/0 overload

      ip nat inside source route-map NAT_ISP2 interface FastEthernet0/1 overload

      ip route 0.0.0.0 0.0.0.0 1.1.1.1

      ip route 0.0.0.0 0.0.0.0 2.2.2.1

      !

      ip access-list standard INSIDE_LIST

      permit 192.168.1.0 0.0.0.255

      !

      route-map NAT_ISP2 permit 10

      match ip address INSIDE_LIST

      match interface FastEthernet0/1

      !

      route-map NAT_ISP1 permit 10

      match ip address INSIDE_LIST

      match interface FastEthernet0/0

      !

      =================================================================================


      R2 Configuration

      =================================================================================

      hostname R2

      !

      interface FastEthernet0/0

      ip address 1.1.1.1 255.255.255.0

      ip access-group BCP38 in

      duplex auto

      speed auto

      !

      interface FastEthernet0/1

      ip address 3.3.3.1 255.255.255.0

      duplex auto

      speed auto

      !

      interface FastEthernet1/0

      ip address 4.4.4.1 255.255.255.0

      duplex auto

      speed auto

      !

      interface FastEthernet1/1

      no ip address

      shutdown

      duplex auto

      speed auto

      !

      router ospf 1

      router-id 100.100.100.100

      passive-interface default

      no passive-interface FastEthernet0/1

      network 1.1.1.1 0.0.0.0 area 0

      network 3.3.3.1 0.0.0.0 area 0

      network 4.4.4.1 0.0.0.0 area 0

      !

      ip access-list standard BCP38

      permit 1.1.1.0 0.0.0.255

      deny   any

      !

      =================================================================================


      R3 Configuration

      =================================================================================

      hostname R3

      !

      interface FastEthernet0/0

      ip address 2.2.2.1 255.255.255.0

      ip access-group BCP38 in

      duplex auto

      speed auto

      !

      interface FastEthernet0/1

      ip address 3.3.3.2 255.255.255.0

      duplex auto

      speed auto

      !

      interface FastEthernet1/0

      ip address 5.5.5.1 255.255.255.0

      duplex auto

      speed auto

      !

      interface FastEthernet1/1

      no ip address

      shutdown

      duplex auto

      speed auto

      !

      router ospf 1

      router-id 200.200.200.200

      passive-interface default

      no passive-interface FastEthernet0/1

      network 2.2.2.1 0.0.0.0 area 0

      network 3.3.3.2 0.0.0.0 area 0

      network 5.5.5.1 0.0.0.0 area 0

      !

      ip access-list standard BCP38

      permit 2.2.2.0 0.0.0.255

      deny   any

      !

      =================================================================================


      Is this an simulation error or bug in Cisco IOS?

      PS I used Cisco IOS 15.2(4)M7 "c7200-adventerprisek9-mz.152-4.M7.bin" image for the simulation.

         
        • 1. Re: Default route ECMP Load Balancing abnormal routing
          Ing_Percy

          Hi

           

          Put the routing table of these devices.

          Apply a "trace" from the PC2 and PC3 to the addresses (1.1.1.2 and 2.2.2.2).

           

          I implemented your topology

           

          The routing table of R3

          cap2.JPG

          Ping from PC3

          cap1.JPG

          On R1, we can see the packets is arriving from 5.5.5.2 (PC-3) to 1.1.1.2 (local: f0/0). After this, the reply from 1.1.1.2 to 5.5.5.2 is sending through interface f0/1 towards R3

          cap3.JPG

          On R3, the packet from 1.1.1.2 to 5.5.5.2 through interface f0/1 is denied because of ACL applied in R3

          cap4.JPG

          R3#sh run int f0/1

          interface FastEthernet0/1

          ip address 2.2.2.1 255.255.255.0

          ip access-group BCP38 in

           

          R3#sh access-list

          Standard IP access list BCP38

              10 permit 2.2.2.0, wildcard bits 0.0.0.255 (62 matches)

              20 deny   any (336 matches)

           

          Regards!

          • 2. Re: Default route ECMP Load Balancing abnormal routing
            Sakar

             

            On R1, we can see the packets is arriving from 5.5.5.2 (PC-3) to 1.1.1.2 (local: f0/0). After this, the reply from 1.1.1.2 to 5.5.5.2 is sending through interface f0/1 towards R3

            cap3.JPG

            On R3, the packet from 1.1.1.2 to 5.5.5.2 through interface f0/1 is denied because of ACL applied in R3

            cap4.JPG

            I think you just confirmed my problem. This is my assumption for the simulation R2 is ISP1 edge router and R3 is ISP2 edge router. ISPs will place source address filtering at their edge router to prevent IP Spoofing by allowing only the Source IPs within the assigned subnet. Consider ISPs are in their own AS. Customer router R1 does not have its own AS and is leasing IPs from both ISPs.

             

            In your simulation case R1 is sending Ping reply with source address of 1.1.1.2 (ISP1 owned IP) to R3(ISP2). ISP2 will drop this traffic.

             

            Ping traffic flow.png

            I think the problem is in R1, R1 should not have chosen (2.2.2.1) R3 as next-hop for reply traffic of echo request for IP (1.1.1.2). If R1 had chosen the same interface (f0/0, 1.1.1.2) on which request traffic was received for reply traffic. The problem would not have occurred.

             

            Since there are two default static routes on R1 probably LB is causing this issue. How to make R1 use the same interface for reply traffic for which request traffic was received while implementing LB?

            • 3. Re: Default route ECMP Load Balancing abnormal routing
              Ing_Percy

              Hi!

               

              Ping traffic flow.png

              One solution is applying PBR

               

              My configuration is the following:

               

              R1(config)#access-list 100 permit ip host 1.1.1.2 4.4.4.0 0.0.0.255

              R1(config)#access-list 100 permit ip host 1.1.1.2 5.5.5.0 0.0.0.255

               

              R1(config)#access-list 150 permit ip host 2.2.2.2 4.4.4.0 0.0.0.255

              R1(config)#access-list 150 permit ip host 2.2.2.2 5.5.5.0 0.0.0.255

               

              R1(config)#route-map PBR1 permit 10

              R1(config-route-map)#match ip address 100

              R1(config-route-map)#set ip next-hop 1.1.1.1

               

              R1(config)#route-map PBR1 permit 20

              R1(config-route-map)#match ip address 150

              R1(config-route-map)#set ip next-hop 2.2.2.1

               

              R1(config)#ip local policy route-map PBR1

               

              In this case, I applied the policy to packets are generated by the same router (R1)

               

              Now, in the PC-2 and PC-3

              ping1.JPG

              ping2.JPG

              Now, R1 use the same interface for reply traffic.

               

              Regards!

              • 4. Re: Default route ECMP Load Balancing abnormal routing
                Martin

                The issue you have is called Asymmetric Routing problem.  it is beyond CCNA scope; more like CCNP and CCIE. you can search for it here or on google.  it  usually comes up when you have multi-home firewalls set up; FHRP like HSRP or VRRP, multihomed BGP in WAN setup.

                 

                lots of stuff about it on you tube Asymmetric vs Symmetric Routing Issue Part 1 - YouTube

                some links here asymmetric routing

                • 5. Re: Default route ECMP Load Balancing abnormal routing
                  Sakar

                  R1(config)#access-list 100 permit ip host 1.1.1.2 4.4.4.0 0.0.0.255

                  R1(config)#access-list 100 permit ip host 1.1.1.2 5.5.5.0 0.0.0.255

                   

                  R1(config)#access-list 150 permit ip host 2.2.2.2 4.4.4.0 0.0.0.255

                  R1(config)#access-list 150 permit ip host 2.2.2.2 5.5.5.0 0.0.0.255

                   

                  R1(config)#route-map PBR1 permit 10

                  R1(config-route-map)#match ip address 100

                  R1(config-route-map)#set ip next-hop 1.1.1.1

                   

                  R1(config)#route-map PBR1 permit 20

                  R1(config-route-map)#match ip address 150

                  R1(config-route-map)#set ip next-hop 2.2.2.1

                  Thank you for your solution but this solves the problem only partially as it only selects two IP subnets from billions of IPs in the internet for PBR.

                   

                  I tried to change the configuration so that it accommodates all the IPs in the internet.

                   

                  =================================================================================

                  ip local policy route-map PBR-RETURN

                  !

                  ip access-list standard ISP1-GW

                  permit 1.1.1.1

                  ip access-list standard ISP1_IP

                  permit 1.1.1.2

                  ip access-list standard ISP2-GW

                  permit 2.2.2.1

                  ip access-list standard ISP2_IP

                  permit 2.2.2.2

                  !

                  route-map PBR-RETURN permit 10

                  match ip address ISP1_IP

                  match ip next-hop ISP2-GW

                  set ip next-hop 1.1.1.1

                  !

                  route-map PBR-RETURN permit 20

                  match ip address ISP2_IP

                  match ip next-hop ISP1-GW

                  set ip next-hop 2.2.2.1

                  !

                  =================================================================================

                   

                  But now the problem is pings from PC-1 and PC-4 (from 192.168.1.0/24 subnet) to 1.1.1.2 and 2.2.2.2 fail. Reply traffic for their pings are also being sent to ISPs.

                  Even though I have two match clauses one to match source IP of Return traffic(1.1.1.2 OR 2.2.2.2), another to match intended next-hop and then set the correct next-hop IP.


                  PBR internal traffic issue PC-1 console.png

                   

                  PBR internal traffic issue R1 console.png

                   

                  How to correct this issue? I am not familiar with route-map or PBR.

                  • 6. Re: Default route ECMP Load Balancing abnormal routing
                    Ing_Percy

                    Hi!

                     

                    The PBR feature is not part of syllabus of CCNA R&S, but the configuration must be the following:

                     

                    R1(config)#ip access-list extended INTERNAL

                    R1(config-ext-nacl)#permit ip host 1.1.1.2 192.168.1.0 0.0.0.255

                    R1(config-ext-nacl)#permit ip host 2.2.2.2 192.168.1.0 0.0.0.255

                     

                    R1(config)#ip access-list standard ISP1

                    R1(config-std-nacl)#permit 1.1.1.2

                     

                    R1(config)#ip access-list standard ISP2

                    R1(config-std-nacl)#permit 2.2.2.2

                     

                    R1(config)#route-map PBR1 permit 5

                    R1(config-route-map)#match ip address INTERNAL

                    R1(config-route-map)#exit

                    R1(config)#route-map PBR1 permit 10

                    R1(config-route-map)#match ip address ISP1

                    R1(config-route-map)#set ip next-hop 1.1.1.1

                    R1(config-route-map)#exit

                    R1(config)#route-map PBR1 permit 20

                    R1(config-route-map)#match ip address ISP2

                    R1(config-route-map)#set ip next-hop 2.2.2.1

                     

                    R1(config)#ip local policy route-map PBR1

                     

                    Now

                     

                    From PC1

                    pbr5.JPG

                    pbr1.JPG

                    On R1

                    pbr2.JPG

                    From PC-2

                    pbr3.JPG

                    From PC-3

                    pbr4.JPG

                     

                    Concepts about PBR

                    Implementing Path Control Using Policy-Based Routing > CCNP ROUTE 642-902 Exam Foundation Learning: Implementing Path Co…

                     

                    Best regards!

                    • 7. Re: Default route ECMP Load Balancing abnormal routing
                      Samer

                      Agree with Martin, asymmetric routing, it is solved with Policy based routing (PBR).

                      • 8. Re: Default route ECMP Load Balancing abnormal routing
                        Juergen Ilse CCNA R&S

                        Sakar schrieb:


                        Thank you for your solution but this solves the problem only partially as it only selects two IP subnets from billions of IPs in the internet for PBR.

                        I don't really see the problem in your case (except for traffic generated by R1 itself). I think, you don't even need route-maps in this case, if R1 is configured with ip loadbalancing per flow (which is default on IOS routers). If traffic from internal network is send to R2, it will be natted behind the interface address of fa0/0, so the return traffic will come back from R2, because R3 has no route via R1 for that ip address. Similar situation for traffic from internal network towards R3: it will be natted behind interface address of fa0/1, so the return traffic will come in via R3, since R2 has no route via R1 for network 2.2.2.0/24 ...

                        Have i overlooked something?

                        • 9. Re: Default route ECMP Load Balancing abnormal routing
                          Sakar

                          jilse-iph

                           

                          I think you have misunderstood here. The pings from internal network to hosts on the internet is fine. The pings from hosts on internet to wan interface of R1 is also fine now. Its the pings from internal network PCs (PC-1 and PC-4) to wan interfaces of R1(1.1.1.2 and 2.2.2.2) that are failing after applying PBR-RETURN route policy.

                          • 10. Re: Default route ECMP Load Balancing abnormal routing
                            Juergen Ilse CCNA R&S

                            In the given scenario, i don't understand, why you need PBR. Access from internal network to the internet should work even without PBR in this case, as packets are natted differently for traffic that goes through R2 and traffic that goes through R3, and the return traffic will therefor go "the right way" even without PBR. Please tell me, what is not woring with your configuration without PBR.

                            • 11. Re: Default route ECMP Load Balancing abnormal routing
                              Sakar

                              Initial issue was not with internal network not being able to access internet hosts. The problem was host on the internet were not able to ping R1's WAN interface as return traffic would sometimes be directed to another ISP.

                               

                              To fix this issue I implemented PBR as suggested by @Ing_Percy (please have a look above PBR-RETURN for implemented policy). Now the issue is internal network cannot ping R1's WAN interface (1.1.1.2 and 2.2.2.2). They are still able to get on the internet.

                              • 12. Re: Default route ECMP Load Balancing abnormal routing
                                Ing_Percy

                                Hi!

                                 

                                Did you try my suggestion in the post number 6?. Now the internal hosts can make ping to all addresses


                                Regards!

                                • 13. Re: Default route ECMP Load Balancing abnormal routing
                                  Sakar

                                  I overlooked INTERNAL ACL and permit 5 route-map in a hurry. That solution would work for only this scenario and is not very scalable and tedious.

                                   

                                  For example what if the topology changes/grows as following.

                                  Scalability.png

                                  Then every time the ACLs and route-map have to be updated for each added subnet.

                                   

                                  The same scenario was being handled by FortiGate very smoothly and efficiently. Since, Cisco is my primary choice I tried to simulate how Cisco would handle such scenario.

                                  • 14. Re: Default route ECMP Load Balancing abnormal routing
                                    Daniel

                                    Hi Sakar,

                                     

                                    I've read through this post and I agree with Juergen. There is absolutely no need for either PBR or any other type of advanced filtering to make things work. ICMP doesn't really care if the return-traffic is "assymmetrical" or not. In fact it's extremely common on internet that you will Transmit path A (R1-R2-R3) and the reply will go in a different path (R3-R1 directly).

                                     

                                    A lot of networks implement load-balancing and it works just that way depending on known routes from routers in each path.

                                     

                                    In your initial topology, to comply with BCP38, your ACL is on the wrong interface.

                                    You can implement this in a couple of different ways, but ISP1 and ISP2 should only not care at all about the source behind R1.

                                    But you are actually filtering networks on the link between R1 and R2 and R1 and R3 which is IMO a wrong implementation to comply with BCP38.

                                     

                                    Reason for that is because in SP-terminology you are filtering between a PE and a PE. If you want to protect your source-networks assigned to your end-customers you should filter this on the CPE which in your topology is the interface directly attached to your hosts. Not on the links between ISPs.

                                     

                                    Also on top of that ISPs in general filter networks the "advertise" to other ISPs. That means they know which networks they own and other ISPs should not be allowed to advertise these networks to them.

                                     

                                    To comply with BCP38 you should filter 4.4.4.0/24 source on R2 and 5.5.5.0/24 source on R3 and 192.168.1.0/24 source on R1.

                                     

                                    That means in order for your ICMP load-balancing from R1 to work it can take both paths via R1-R2 or R1-R3-R2 assuming all routers know about each networks in your topology.

                                     

                                    But when you filter on the links between ISPs you create a problem. These links should typically just filter routing-advertisements between AS-domains. They are transport suppliers between point A and B and should not care at all whether the source-traffic is 192.168.1.0/24, 4.4.4.0/24 or 5.5.5.0/24.

                                     

                                    This is a general recommendation and various topologies of course have different needs and problems to address.

                                     

                                    "Initial issue was not with internal network not being able to access internet hosts. The problem was host on the internet were not able to ping R1's WAN interface as return traffic would sometimes be directed to another ISP"

                                     

                                     

                                    Correct with your filtering/acls this is what happens. However this is not the recommended way to filter source-traffic to comply with BCP38. If you would filter source-addresses on the correct side of the edge this would not happen. But when you filter inside the transit-networks things like this might happen.

                                     

                                    There is absolutely no need for any PBR at all in your topology to comply with BCP38...working around your ACL-filtering with PBR is just working around a problem that you created that should not exist due to wrong ACL-implementation.

                                     

                                    Note that "wrong" in this case means that you are not filtering source-addresses assigned at the edge. You are filtering networks on a transit-link where we typically want to filter network advertisements and not edge-traffic. That means imagine R1, R2, R3 running BGP with 3 different AS-numbers using the above topology. Then on the transit-links you filter routing-advertisements for security.

                                     

                                    So to answer your latest question - just put your ACL-filtering source networks facing your edge and you never have to change it as long as your IP-addressing plan doesn't change in that customer segment!

                                     

                                    Whatever happens between your various customer-edges is not your SP/Transit-networks responsibility.

                                     

                                    -HTH

                                    Daniel

                                    1 2 Previous Next