3 Replies Latest reply: Apr 7, 2019 2:16 AM by erkanbal RSS

    ACI transit routing... still not working

    Micheline

      Hello CLN and I hope everyone is having a happy and safe end of the year holiday. 

       

      I'm still trying to make this transit routing through an ACI fabric work.  Here's the setup.  Screen Shot 2018-12-28 at 3.53.22 PM.png

      There are two L3Outs, in the same tenant and VRF.  The L3Out on port-15 uses Loopback IP address 3.0.0.111 to peer to CORE1, which is 3.0.0.102.  The L3Out on port-45 uses Loopback IP address 3.0.0.101 to peer with pod3-ext1, which is 3.0.0.104.  Both eBGP peers are based on a static route to the appropriate loopback IP address, and both peerings have come up to ESTABLISHED.

       

      CORE1 has a host, 10.1.0.1/32, that it advertises to the ACI fabric.  ext1 has a second host, 10.1.0.2/32, that it also advertises to the ACI fabric.The external hosts can both be pinged by a VM attached to the ACI fabric.  The fabric itself generates a summary route for the VMs attached to the ACI fabric.  That summary route is received by both external devices.

       

      Here's my problem.  The ACI fabric has not passed the host routes along to the remote device.  That is, CORE1 is not getting the route to 10.1.0.2 and ext1 is not getting the route to 10.1.0.1.  Each L3Out is configured with the following subnets:

       

      10.1.0.1/32         External Subnets for External EGP

      10.1.0.2/32         External Subnets for External EGP

      172.23.0.0/16     External Subnets for External EPG

                                 Export Route Control Subnet

                                 default BGP Route Summarization policy

      172.25.1.0/29     External Subnets for External EPG

                                 Export Route Control Subnet

       

       

      Here's the BGP table for each device.

       

      pod3-ext-1# sh ip bgp

      BGP routing table information for VRF default, address family IPv4 Unicast

      BGP table version is 25, Local Router ID is 3.0.0.104

      Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best

      Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-inject

      ed

      Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup

       

         Network            Next Hop            Metric     LocPrf     Weight Path

      *>l10.1.0.2/32        0.0.0.0                           100      32768 i

      *>e172.23.0.0/16      3.0.0.101                                      0 301 300 i

       

       

      pod3-leaf101# show bgp ipv4 unicast vrf Bluefish:VRF1

      BGP routing table information for VRF Bluefish:VRF1, address family IPv4 Unicast

      BGP table version is 207, local router ID is 3.0.0.101

      Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best

      Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected

      Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup

       

         Network            Next Hop            Metric     LocPrf     Weight Path

      *>r3.0.0.101/32       0.0.0.0                  0        100      32768 ?

      *>r3.0.0.102/32       0.0.0.0                  0        100      32768 ?

      *>r3.0.0.104/32       0.0.0.0                  0        100      32768 ?

      *>r3.0.0.111/32       0.0.0.0                  0        100      32768 ?

      *>e10.1.0.1/32        3.0.0.102                                      0 301 500 i

      *>e10.1.0.2/32        3.0.0.104                                      0 301 3001 i

      *>a172.23.0.0/16      0.0.0.0                           100      32768 i

      s>r172.23.1.0/24      0.0.0.0                  0        100      32768 ?

      s>r172.23.2.0/24      0.0.0.0                  0        100      32768 ?

      *>r172.25.1.2/31      0.0.0.0                  0        100      32768 ?

      *>r172.25.1.4/31      0.0.0.0                  0        100      32768 ?

       

      CORE-1(config-if)# sh ip bgp vrf Bluefish:VRF1

      BGP routing table information for VRF Bluefish:VRF1, address family IPv4 Unicast

      BGP table version is 10, Local Router ID is 3.0.0.102

      Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best

      Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-inject

      ed

      Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup

       

         Network            Next Hop            Metric     LocPrf     Weight Path

      *>l10.1.0.1/32        0.0.0.0                           100      32768 i

      *>e172.23.0.0/16      3.0.0.111                                      0 301 300 i

       

       

      I can see that the ACI fabric is receiving both routes, but neither remote peer is getting the other's host route.  I've tried Export Route Control Subnet on the host routes, Shared Security Import Subnet on the host routes, and entering the transit subnet as two individual subnets (that is, 172.25.1.2/31 on the L3Out on port-15 and 172.25.1.4/31 on the L3Out on port-45).  But none of those combinations work to pass the host routes to the remote device.

       

      Any help would be appreciated.  Thank you in advance!  MM

         
        • 1. Re: ACI transit routing... still not working
          Toshi Nakano

          Hi Micheline,

           

          If you still have the config, can you attach the both L3OUT EPG's config? I am thinking if the config is correct, maybe we can not mix with the route summarization option? What will happen, remove the route summarization?

           

          I think you need to do the following.

           

          L3OUT EPG1 for Ext 1

          10.1.0.1/32        

          External Subnets for External EGP

          Export Route Control Subnet

           

          L3OUT EPG2 for Core 1

          10.1.0.2/32        

          External Subnets for External EGP

          Export Route Control Subnet                          

           

          If the above didn't work .For troubleshooting, what will happen, if you do export all transit subnets.

           

          L3OUT EPG1 for Ext 1

          0.0.0.0/0

          External Subnets for External EGP

          Export Route Control Subnet

          Aggregate Export

           

          L3OUT EPG1 for Core 1

          0.0.0.0/0

          External Subnets for External EGP

          Export Route Control Subnet

          Aggregate Export

          • 2. Re: ACI transit routing... still not working
            Micheline

            OK, success comes to those who are persistent AF.  Thank you to Toshi and everyone else who helped out with this problem.

             

            I *finally* got transit routing to work.  It is NOT the configuration as documented in Stuart Fordham's Cisco ACI Cookbook, BTW.  First of all, I got rid of the route summarization to make troubleshooting easier.  Then I selected "external subnets for external EPG" and "export route control subnet" for the remote subnets AND the transit subnet.  (The ACI Cookbook configuration is export route control only on the transit subnet, and only external subnets for external EPG on the remote subnets.)

             

            Going to the remote devices, I get the host routes shared.

             

            CORE-1# sh ip route vrf Bluefish:VRF1

            IP Route Table for VRF "Bluefish:VRF1"

            '*' denotes best ucast next-hop

            '**' denotes best mcast next-hop

            '[x/y]' denotes [preference/metric]

            '%<string>' in via output denotes VRF <string>

             

            3.0.0.102/32, ubest/mbest: 2/0, attached

                *via 3.0.0.102, Lo301, [0/0], 4d19h, local

                *via 3.0.0.102, Lo301, [0/0], 4d19h, direct

            3.0.0.111/32, ubest/mbest: 1/0

                *via 172.25.1.2, [1/0], 4d19h, static

            10.1.0.1/32, ubest/mbest: 2/0, attached

                *via 10.1.0.1, Lo300, [0/0], 4d19h, local

                *via 10.1.0.1, Lo300, [0/0], 4d19h, direct

            10.1.0.2/32, ubest/mbest: 1/0

                *via 3.0.0.111, [20/0], 00:05:45, bgp-500, external, tag 301

            172.23.1.0/24, ubest/mbest: 1/0

                *via 3.0.0.111, [20/0], 00:16:53, bgp-500, external, tag 301

            172.25.1.2/31, ubest/mbest: 1/0, attached

                *via 172.25.1.3, Eth3/9, [0/0], 4d20h, direct

            172.25.1.3/32, ubest/mbest: 1/0, attached

                *via 172.25.1.3, Eth3/9, [0/0], 4d20h, local

             

            pod3-ext-1# sh ip route

            IP Route Table for VRF "default"

            '*' denotes best ucast next-hop

            '**' denotes best mcast next-hop

            '[x/y]' denotes [preference/metric]

            '%<string>' in via output denotes VRF <string>

             

            3.0.0.101/32, ubest/mbest: 1/0

                *via 172.25.1.4, [1/0], 4d19h, static

            3.0.0.104/31, ubest/mbest: 1/0, attached

                *via 3.0.0.104, Lo3, [0/0], 3w5d, direct

            3.0.0.104/32, ubest/mbest: 1/0, attached

                *via 3.0.0.104, Lo3, [0/0], 3w5d, local

            10.1.0.1/32, ubest/mbest: 1/0

                *via 3.0.0.101, [20/0], 00:02:47, bgp-3001, external, tag 301

            10.1.0.2/32, ubest/mbest: 2/0, attached

                *via 10.1.0.2, Lo100, [0/0], 4d19h, local

                *via 10.1.0.2, Lo100, [0/0], 4d19h, direct

            172.23.2.0/24, ubest/mbest: 1/0

                *via 3.0.0.101, [20/0], 00:16:54, bgp-3001, external, tag 301

            172.25.1.4/31, ubest/mbest: 1/0, attached

                *via 172.25.1.5, Eth3/45, [0/0], 4d22h, direct

            172.25.1.5/32, ubest/mbest: 1/0, attached

                *via 172.25.1.5, Eth3/45, [0/0], 4d22h, local

             

            A host-to-host ping now crosses the ACI fabric and returns successfully.

            pod3-ext-1# ping

            Vrf context to use [default] :

            No user input: using default context

            Target IP address or Hostname: 10.1.0.1

            Repeat count [5] :

            Packet-size [56] : 9000

            Timeout in seconds [2] :

            Sending interval in seconds [0] :

            Extended commands [no] : y

            Source address or interface : 10.1.0.2

            Data pattern [0xabcd] :

            Type of service [0] :

            Set DF bit in IP header [no] :

            Time to live [255] :

            Loose, Strict, Record, Timestamp, Verbose [None] :

            Sweep range of sizes [no] :

            Sending 5, 9000-bytes ICMP Echos to 10.1.0.1

            Timeout is 2 seconds, data pattern is 0xABCD

             

            9008 bytes from 10.1.0.1: icmp_seq=0 ttl=252 time=1.594 ms

            9008 bytes from 10.1.0.1: icmp_seq=1 ttl=252 time=1.587 ms

            .9008 bytes from 10.1.0.1: icmp_seq=3 ttl=252 time=1.822 ms

            9008 bytes from 10.1.0.1: icmp_seq=4 ttl=252 time=1.332 ms

             

            --- 10.1.0.1 ping statistics ---

            5 packets transmitted, 4 packets received, 20.00% packet loss

            round-trip min/avg/max = 1.332/1.583/1.822 ms

             

            For kicks and giggles, I wanted to see if it was the route summarization or the subnet settings that were nixing up the works.  After experimentation, the route summarization was ruled out as the problem.  Notice below, that the remote devices' RIBs have both host routes AND the ACI summarized route.

             

            CORE-1# sh ip route vrf Bluefish:VRF1

            IP Route Table for VRF "Bluefish:VRF1"

            '*' denotes best ucast next-hop

            '**' denotes best mcast next-hop

            '[x/y]' denotes [preference/metric]

            '%<string>' in via output denotes VRF <string>

             

            3.0.0.102/32, ubest/mbest: 2/0, attached

                *via 3.0.0.102, Lo301, [0/0], 4d19h, local

                *via 3.0.0.102, Lo301, [0/0], 4d19h, direct

            3.0.0.111/32, ubest/mbest: 1/0

                *via 172.25.1.2, [1/0], 4d19h, static

            10.1.0.1/32, ubest/mbest: 2/0, attached

                *via 10.1.0.1, Lo300, [0/0], 4d19h, local

                *via 10.1.0.1, Lo300, [0/0], 4d19h, direct

            10.1.0.2/32, ubest/mbest: 1/0

                *via 3.0.0.111, [20/0], 00:19:47, bgp-500, external, tag 301

            172.23.0.0/16, ubest/mbest: 1/0

                *via 3.0.0.111, [20/0], 00:00:03, bgp-500, external, tag 301

            172.25.1.2/31, ubest/mbest: 1/0, attached

                *via 172.25.1.3, Eth3/9, [0/0], 4d21h, direct

            172.25.1.3/32, ubest/mbest: 1/0, attached

                *via 172.25.1.3, Eth3/9, [0/0], 4d21h, local

             

            pod3-ext-1# sh ip route

            IP Route Table for VRF "default"

            '*' denotes best ucast next-hop

            '**' denotes best mcast next-hop

            '[x/y]' denotes [preference/metric]

            '%<string>' in via output denotes VRF <string>

             

            3.0.0.101/32, ubest/mbest: 1/0

                *via 172.25.1.4, [1/0], 4d19h, static

            3.0.0.104/31, ubest/mbest: 1/0, attached

                *via 3.0.0.104, Lo3, [0/0], 3w5d, direct

            3.0.0.104/32, ubest/mbest: 1/0, attached

                *via 3.0.0.104, Lo3, [0/0], 3w5d, local

            10.1.0.1/32, ubest/mbest: 1/0

                *via 3.0.0.101, [20/0], 00:14:10, bgp-3001, external, tag 301

            10.1.0.2/32, ubest/mbest: 2/0, attached

                *via 10.1.0.2, Lo100, [0/0], 4d19h, local

                *via 10.1.0.2, Lo100, [0/0], 4d19h, direct

            172.23.0.0/16, ubest/mbest: 1/0

                *via 3.0.0.101, [20/0], 00:00:04, bgp-3001, external, tag 301

            172.25.1.4/31, ubest/mbest: 1/0, attached

                *via 172.25.1.5, Eth3/45, [0/0], 4d23h, direct

            172.25.1.5/32, ubest/mbest: 1/0, attached

                *via 172.25.1.5, Eth3/45, [0/0], 4d23h, local

             

            What causes the break is when the remote subnets are not configured with both external subnet for external EPG and export route control subnet.

             

            Thanks again to everyone who chimed in.  MM

            • 3. Re: ACI transit routing... still not working
              erkanbal

              Great post Michelins Thanks..