10 Replies Latest reply: May 19, 2019 5:23 AM by Juergen Ilse CCNA R&S RSS

    Default route ECMP Load Balancing abnormal routing

    Sakar

      How does ECMP Load Balancing algorithm work for incoming traffic?

       

      I tried to simulate the scenario in GNS3 and have got varying results.

      Here is my topology.

       

      Topology.png

       

      R1 is customer router, R2 is ISP1, R3 is ISP2, PC-2 and PC-3 are hosts on the internet.

      Assume 1.1.1.0 /24 and 4.4.4.0 /24 blocks owned by R2 (ISP1). 2.2.2.0 /24 and 5.5.5.0 /24 blocks owned by R3 (ISP2). 3.3.3.0 /24 block can be assumed to be owned by either ISP.


      R1 has two default static routes; one to ISP1 another to ISP2.

       

      ISPs will generally employ BCP38 (filtering source address, allowing source IPs within their network at the edge router). Here, I have created standard ACL on R2 allowing only 1.1.1.0 /24 network source IPs to ingress through f0/0 interface (1.1.1.1). Similar configuration on R3 allowing only 2.2.2.0 /24 network source IPs to ingress through f0/0 interface (2.2.2.1).

       

      Because of BCP38 R1 should forward return traffic of incoming traffic from internet through the same interface in which it was received. If it chooses interface connected to another ISP (maybe due to LB logic), the other ISP will drop the traffic.

       

      This is what I expected. But for multiple simulation runs I have obtained different results.

       

      Case I: R1 routes return traffic from the same interface on which request traffic was received (as per my expectation) making pings from PC-2 and PC-3 to R1's public IP 1.1.1.2 and 2.2.2.2 successful.

       

      Case 2: Pings from PC-2 to R1's public IP 1.1.1.2 and 2.2.2.2 are successful but pings from PC-3 to R1's public IP 1.1.1.2 and 2.2.2.2 are unsuccessful. Pings for 1.1.1.2 are unsuccessful because Ping reply for IP 1.1.1.2 is forwarded to R3 (2.2.2.1) which is dropped by R3. Similarly, Pings for 2.2.2.2 are unsuccessful because Ping reply for IP 2.2.2.2 is forwarded to R2 (1.1.1.1) which is also dropped.

       

      Case 3: Vice-Versa case of case 2 pings from PC-3 are successful while pings from PC-2 are unsuccessful.

       

      Case 4: Pings from both PC-2 and PC-3 fail.

       

      Here are screenshots for Case 3.

       

      PC2 ping R1; PC2 console.png

      Pings from PC-2 fail

       

      PC2 ping R1; R1 console.png

      Debug messages on R1 when pings from PC-2 failed

       

      PC3 ping R1; PC3 console.png

      Pings from PC-3 are successful

       

      PC3 ping R1; R1 console.png

      Debug messages on R1 when pings from PC-3 are successful

       

      R1 routes.png

      Default Routes on R1

       

      R1 Configuration

      =================================================================================

      hostname R1

      !

      interface FastEthernet0/0

      ip address 1.1.1.2 255.255.255.0

      ip nat outside

      ip virtual-reassembly in

      duplex auto

      speed auto

      !

      interface FastEthernet0/1

      ip address 2.2.2.2 255.255.255.0

      ip nat outside

      ip virtual-reassembly in

      duplex auto

      speed auto

      !

      interface FastEthernet1/0

      ip address 192.168.1.1 255.255.255.0

      ip nat inside

      ip virtual-reassembly in

      duplex auto

      speed auto

      !

      interface FastEthernet1/1

      no ip address

      shutdown

      duplex auto

      speed auto

      !

      ip nat inside source route-map NAT_ISP1 interface FastEthernet0/0 overload

      ip nat inside source route-map NAT_ISP2 interface FastEthernet0/1 overload

      ip route 0.0.0.0 0.0.0.0 1.1.1.1

      ip route 0.0.0.0 0.0.0.0 2.2.2.1

      !

      ip access-list standard INSIDE_LIST

      permit 192.168.1.0 0.0.0.255

      !

      route-map NAT_ISP2 permit 10

      match ip address INSIDE_LIST

      match interface FastEthernet0/1

      !

      route-map NAT_ISP1 permit 10

      match ip address INSIDE_LIST

      match interface FastEthernet0/0

      !

      =================================================================================


      R2 Configuration

      =================================================================================

      hostname R2

      !

      interface FastEthernet0/0

      ip address 1.1.1.1 255.255.255.0

      ip access-group BCP38 in

      duplex auto

      speed auto

      !

      interface FastEthernet0/1

      ip address 3.3.3.1 255.255.255.0

      duplex auto

      speed auto

      !

      interface FastEthernet1/0

      ip address 4.4.4.1 255.255.255.0

      duplex auto

      speed auto

      !

      interface FastEthernet1/1

      no ip address

      shutdown

      duplex auto

      speed auto

      !

      router ospf 1

      router-id 100.100.100.100

      passive-interface default

      no passive-interface FastEthernet0/1

      network 1.1.1.1 0.0.0.0 area 0

      network 3.3.3.1 0.0.0.0 area 0

      network 4.4.4.1 0.0.0.0 area 0

      !

      ip access-list standard BCP38

      permit 1.1.1.0 0.0.0.255

      deny   any

      !

      =================================================================================


      R3 Configuration

      =================================================================================

      hostname R3

      !

      interface FastEthernet0/0

      ip address 2.2.2.1 255.255.255.0

      ip access-group BCP38 in

      duplex auto

      speed auto

      !

      interface FastEthernet0/1

      ip address 3.3.3.2 255.255.255.0

      duplex auto

      speed auto

      !

      interface FastEthernet1/0

      ip address 5.5.5.1 255.255.255.0

      duplex auto

      speed auto

      !

      interface FastEthernet1/1

      no ip address

      shutdown

      duplex auto

      speed auto

      !

      router ospf 1

      router-id 200.200.200.200

      passive-interface default

      no passive-interface FastEthernet0/1

      network 2.2.2.1 0.0.0.0 area 0

      network 3.3.3.2 0.0.0.0 area 0

      network 5.5.5.1 0.0.0.0 area 0

      !

      ip access-list standard BCP38

      permit 2.2.2.0 0.0.0.255

      deny   any

      !

      =================================================================================


      Is this an simulation error or bug in Cisco IOS?

      PS I used Cisco IOS 15.2(4)M7 "c7200-adventerprisek9-mz.152-4.M7.bin" image for the simulation.

         
        • 1. Re: Default route ECMP Load Balancing abnormal routing
          Ing_Percy

          Hi

           

          Put the routing table of these devices.

          Apply a "trace" from the PC2 and PC3 to the addresses (1.1.1.2 and 2.2.2.2).

           

          I implemented your topology

           

          The routing table of R3

          cap2.JPG

          Ping from PC3

          cap1.JPG

          On R1, we can see the packets is arriving from 5.5.5.2 (PC-3) to 1.1.1.2 (local: f0/0). After this, the reply from 1.1.1.2 to 5.5.5.2 is sending through interface f0/1 towards R3

          cap3.JPG

          On R3, the packet from 1.1.1.2 to 5.5.5.2 through interface f0/1 is denied because of ACL applied in R3

          cap4.JPG

          R3#sh run int f0/1

          interface FastEthernet0/1

          ip address 2.2.2.1 255.255.255.0

          ip access-group BCP38 in

           

          R3#sh access-list

          Standard IP access list BCP38

              10 permit 2.2.2.0, wildcard bits 0.0.0.255 (62 matches)

              20 deny   any (336 matches)

           

          Regards!

          • 2. Re: Default route ECMP Load Balancing abnormal routing
            Sakar

             

            On R1, we can see the packets is arriving from 5.5.5.2 (PC-3) to 1.1.1.2 (local: f0/0). After this, the reply from 1.1.1.2 to 5.5.5.2 is sending through interface f0/1 towards R3

            cap3.JPG

            On R3, the packet from 1.1.1.2 to 5.5.5.2 through interface f0/1 is denied because of ACL applied in R3

            cap4.JPG

            I think you just confirmed my problem. This is my assumption for the simulation R2 is ISP1 edge router and R3 is ISP2 edge router. ISPs will place source address filtering at their edge router to prevent IP Spoofing by allowing only the Source IPs within the assigned subnet. Consider ISPs are in their own AS. Customer router R1 does not have its own AS and is leasing IPs from both ISPs.

             

            In your simulation case R1 is sending Ping reply with source address of 1.1.1.2 (ISP1 owned IP) to R3(ISP2). ISP2 will drop this traffic.

             

            Ping traffic flow.png

            I think the problem is in R1, R1 should not have chosen (2.2.2.1) R3 as next-hop for reply traffic of echo request for IP (1.1.1.2). If R1 had chosen the same interface (f0/0, 1.1.1.2) on which request traffic was received for reply traffic. The problem would not have occurred.

             

            Since there are two default static routes on R1 probably LB is causing this issue. How to make R1 use the same interface for reply traffic for which request traffic was received while implementing LB?

            • 3. Re: Default route ECMP Load Balancing abnormal routing
              Ing_Percy

              Hi!

               

              Ping traffic flow.png

              One solution is applying PBR

               

              My configuration is the following:

               

              R1(config)#access-list 100 permit ip host 1.1.1.2 4.4.4.0 0.0.0.255

              R1(config)#access-list 100 permit ip host 1.1.1.2 5.5.5.0 0.0.0.255

               

              R1(config)#access-list 150 permit ip host 2.2.2.2 4.4.4.0 0.0.0.255

              R1(config)#access-list 150 permit ip host 2.2.2.2 5.5.5.0 0.0.0.255

               

              R1(config)#route-map PBR1 permit 10

              R1(config-route-map)#match ip address 100

              R1(config-route-map)#set ip next-hop 1.1.1.1

               

              R1(config)#route-map PBR1 permit 20

              R1(config-route-map)#match ip address 150

              R1(config-route-map)#set ip next-hop 2.2.2.1

               

              R1(config)#ip local policy route-map PBR1

               

              In this case, I applied the policy to packets are generated by the same router (R1)

               

              Now, in the PC-2 and PC-3

              ping1.JPG

              ping2.JPG

              Now, R1 use the same interface for reply traffic.

               

              Regards!

              • 4. Re: Default route ECMP Load Balancing abnormal routing
                Martin

                The issue you have is called Asymmetric Routing problem.  it is beyond CCNA scope; more like CCNP and CCIE. you can search for it here or on google.  it  usually comes up when you have multi-home firewalls set up; FHRP like HSRP or VRRP, multihomed BGP in WAN setup.

                 

                lots of stuff about it on you tube Asymmetric vs Symmetric Routing Issue Part 1 - YouTube

                some links here asymmetric routing

                • 5. Re: Default route ECMP Load Balancing abnormal routing
                  Sakar

                  R1(config)#access-list 100 permit ip host 1.1.1.2 4.4.4.0 0.0.0.255

                  R1(config)#access-list 100 permit ip host 1.1.1.2 5.5.5.0 0.0.0.255

                   

                  R1(config)#access-list 150 permit ip host 2.2.2.2 4.4.4.0 0.0.0.255

                  R1(config)#access-list 150 permit ip host 2.2.2.2 5.5.5.0 0.0.0.255

                   

                  R1(config)#route-map PBR1 permit 10

                  R1(config-route-map)#match ip address 100

                  R1(config-route-map)#set ip next-hop 1.1.1.1

                   

                  R1(config)#route-map PBR1 permit 20

                  R1(config-route-map)#match ip address 150

                  R1(config-route-map)#set ip next-hop 2.2.2.1

                  Thank you for your solution but this solves the problem only partially as it only selects two IP subnets from billions of IPs in the internet for PBR.

                   

                  I tried to change the configuration so that it accommodates all the IPs in the internet.

                   

                  =================================================================================

                  ip local policy route-map PBR-RETURN

                  !

                  ip access-list standard ISP1-GW

                  permit 1.1.1.1

                  ip access-list standard ISP1_IP

                  permit 1.1.1.2

                  ip access-list standard ISP2-GW

                  permit 2.2.2.1

                  ip access-list standard ISP2_IP

                  permit 2.2.2.2

                  !

                  route-map PBR-RETURN permit 10

                  match ip address ISP1_IP

                  match ip next-hop ISP2-GW

                  set ip next-hop 1.1.1.1

                  !

                  route-map PBR-RETURN permit 20

                  match ip address ISP2_IP

                  match ip next-hop ISP1-GW

                  set ip next-hop 2.2.2.1

                  !

                  =================================================================================

                   

                  But now the problem is pings from PC-1 and PC-4 (from 192.168.1.0/24 subnet) to 1.1.1.2 and 2.2.2.2 fail. Reply traffic for their pings are also being sent to ISPs.

                  Even though I have two match clauses one to match source IP of Return traffic(1.1.1.2 OR 2.2.2.2), another to match intended next-hop and then set the correct next-hop IP.


                  PBR internal traffic issue PC-1 console.png

                   

                  PBR internal traffic issue R1 console.png

                   

                  How to correct this issue? I am not familiar with route-map or PBR.

                  • 6. Re: Default route ECMP Load Balancing abnormal routing
                    Ing_Percy

                    Hi!

                     

                    The PBR feature is not part of syllabus of CCNA R&S, but the configuration must be the following:

                     

                    R1(config)#ip access-list extended INTERNAL

                    R1(config-ext-nacl)#permit ip host 1.1.1.2 192.168.1.0 0.0.0.255

                    R1(config-ext-nacl)#permit ip host 2.2.2.2 192.168.1.0 0.0.0.255

                     

                    R1(config)#ip access-list standard ISP1

                    R1(config-std-nacl)#permit 1.1.1.2

                     

                    R1(config)#ip access-list standard ISP2

                    R1(config-std-nacl)#permit 2.2.2.2

                     

                    R1(config)#route-map PBR1 permit 5

                    R1(config-route-map)#match ip address INTERNAL

                    R1(config-route-map)#exit

                    R1(config)#route-map PBR1 permit 10

                    R1(config-route-map)#match ip address ISP1

                    R1(config-route-map)#set ip next-hop 1.1.1.1

                    R1(config-route-map)#exit

                    R1(config)#route-map PBR1 permit 20

                    R1(config-route-map)#match ip address ISP2

                    R1(config-route-map)#set ip next-hop 2.2.2.1

                     

                    R1(config)#ip local policy route-map PBR1

                     

                    Now

                     

                    From PC1

                    pbr5.JPG

                    pbr1.JPG

                    On R1

                    pbr2.JPG

                    From PC-2

                    pbr3.JPG

                    From PC-3

                    pbr4.JPG

                     

                    Concepts about PBR

                    Implementing Path Control Using Policy-Based Routing > CCNP ROUTE 642-902 Exam Foundation Learning: Implementing Path Co…

                     

                    Best regards!

                    • 7. Re: Default route ECMP Load Balancing abnormal routing
                      Samer

                      Agree with Martin, asymmetric routing, it is solved with Policy based routing (PBR).

                      • 8. Re: Default route ECMP Load Balancing abnormal routing
                        Juergen Ilse CCNA R&S

                        Sakar schrieb:


                        Thank you for your solution but this solves the problem only partially as it only selects two IP subnets from billions of IPs in the internet for PBR.

                        I don't really see the problem in your case (except for traffic generated by R1 itself). I think, you don't even need route-maps in this case, if R1 is configured with ip loadbalancing per flow (which is default on IOS routers). If traffic from internal network is send to R2, it will be natted behind the interface address of fa0/0, so the return traffic will come back from R2, because R3 has no route via R1 for that ip address. Similar situation for traffic from internal network towards R3: it will be natted behind interface address of fa0/1, so the return traffic will come in via R3, since R2 has no route via R1 for network 2.2.2.0/24 ...

                        Have i overlooked something?

                        • 9. Re: Default route ECMP Load Balancing abnormal routing
                          Sakar

                          jilse-iph

                           

                          I think you have misunderstood here. The pings from internal network to hosts on the internet is fine. The pings from hosts on internet to wan interface of R1 is also fine now. Its the pings from internal network PCs (PC-1 and PC-4) to wan interfaces of R1(1.1.1.2 and 2.2.2.2) that are failing after applying PBR-RETURN route policy.

                          • 10. Re: Default route ECMP Load Balancing abnormal routing
                            Juergen Ilse CCNA R&S

                            In the given scenario, i don't understand, why you need PBR. Access from internal network to the internet should work even without PBR in this case, as packets are natted differently for traffic that goes through R2 and traffic that goes through R3, and the return traffic will therefor go "the right way" even without PBR. Please tell me, what is not woring with your configuration without PBR.