7 Replies Latest reply: Aug 30, 2019 7:35 AM by Elvin Arias RSS

    The Adventures of Layer One Larry (#1):  I pinged what?

    Micheline

      The Adventures of Layer One Larry (#1): I pinged what? 

      by Micheline Murphy

       

      It’s that time of year again, summer vacation.  You wisely chose to go somewhere with spotty Internet coverage... because you know that given half the chance, your somewhat less than reliable co-worker, Larry, will call/text/email you to help him with some problem or other in the data center that you both are assigned to.

       

      This year’s summer vacation is to central British Columbia, where you taking in Mother Nature’s splendid glory on the loop between Prince George and Kamloops in your RV.  And, ah!  No bars = blessed peace.  Sadly, you make the mistake of checking your email while in an Internet cafe in Prince George and see this email from Larry.

       

      Hey buddy—I know you’re on vacation, but I’m in a bit of a bind and could really use some help on this.  I was cleaning up the config a bit, and now it seems like the old network can’t reach the WAN router anymore, even though the WAN router can still ping into the old network. 

      -Larry

       

      Attached is this snippet that Larry cut from the CLI.

       

      WAN_router#ping vrf Twofish 2.2.2.2

      Type escape sequence to abort.

      Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:

      !!!!!

      Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

       

      old_network#ping vrf Twofish 1.1.1.1

      Type escape sequence to abort.

      Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:

      .....

      Success rate is 0 percent (0/5)

       

       

      NOOOOO!  After pounding your head on the keyboard and gritting your teeth enough to undo all the rest and relaxation of your vacation, you take a moment to think about the problem. 

       

      You remember that the old network is linked to Leaf 101 of your brand new ACI fabric by L3Out-a.  The WAN router is linked to Leaf 102 of the same fabric using L3Out-b.  You also know that the old network could ping the WAN router before you left, because you, yourself, set it up.  Fortunately/ Unfortunately, you can’t get into the APIC GUI even though it’s web-based because company security policies limit access to company-issued laptops, which you left at the office... because, vacation.

       

      What do you tell Larry?

       

      The Rules:

      Private message me with your answer.  Do not post your answer here.  I’d like to let everyone have a crack at solving this beforehand.  After you have PM’ed me, post a comment here with your favorite vacation destination. The first person to PM me with the right answer will be awarded “correct.”  Any person after that will get a “helpful.”  I will post the official answer in a week. 

       

      Good luck! 

        • 1. Re: The Adventures of Layer One Larry (#1):  I pinged what?
          Juergen Ilse CCNA R&S

          Fortunately, i don't work at the same company than Larry ...

          • 2. Re: The Adventures of Layer One Larry (#1):  I pinged what?
            Jerome Tissieres

            Hi Micheline,

             

            Thank you for creating this nice scenario.

             

            My favorite holidays destination is the Seychelles.

            https://www.seychelles.travel/en/

             

            I sent you my answers in PM.

             

            Have a good day,

            Jerome

            • 3. Re: The Adventures of Layer One Larry (#1):  I pinged what?
              Micheline

              Hello DC Troubleshooters!  Seeing as the responses to my original post about a Tshoot challenge was good, but spread out over a few weeks, I'm going to delay posting the answer until next week on Monday!

               

              Put your thinking caps on!  MM

              • 4. Re: The Adventures of Layer One Larry (#1):  I pinged what?
                Micheline

                The Answer:

                First, let’s get some bearings on the topology.  Whenever a design calls for traffic to enter an ACI fabric and egress the fabric en route to its final destination, we’re talking about transit routing.  The topology looks like this:

                 

                 

                 

                Screen Shot 2019-08-22 at 9.15.50 AM.png

                 

                Transit routing requires a few things:

                • Two L3Outs, typically with some sort of routing protocol configured between the ACI border leaf and the non-ACI peer device
                • External EPGs configured with the proper subnets and with the proper flags on each subnet
                • A contract between the two external EPGs

                 

                The two pings that Larry sent us don’t tell us much, but we can deduce a few things.  First, we know that the contract appears to have been configured correctly because traffic is passing back and forth between the old_network L3Out and the WAN_router.  We also know that (at least some) external routes have been passed between the two border leaf switches.

                 

                But let’s take a closer look at the information Larry sent us.  He was on the ball enough to make sure that his ping was using the right VRF, but he used a basic ping.  Recall that the basic ping uses the IP address of the egress port as the source IP. In order for a remote device to return a basic ping, it needs to have routing information for that IP.  In this scenario, that means that the ping that failed had a source IP address of 172.30.1.5. 

                 

                Does the WAN_router have information enough to return a ping to that address?  Let’s look.

                 

                WAN_router#sh ip route vrf Twofish 172.30.1.5 

                 

                Routing Table: Twofish

                % Subnet not in table

                 

                No, it does not.  So, it is no wonder that a ping from the old_network to the WAN_router is failing.  But that’s with a basic ping.  What happens if we get more specific and drill down on our ping parameters?

                 

                WAN_router#ping vrf Twofish

                Protocol [ip]:

                Target IP address: 2.2.2.2

                Repeat count [5]:

                Datagram size [100]:

                Timeout in seconds [2]:

                Extended commands [n]: y

                Ingress ping [n]:

                Source address or interface: 1.1.1.1

                Type of service [0]:

                Set DF bit in IP header? [no]:

                Validate reply data? [no]:

                Data pattern [0x0000ABCD]:

                Loose, Strict, Record, Timestamp, Verbose[none]:

                Sweep range of sizes [n]:

                Type escape sequence to abort.

                Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:

                Packet sent with a source address of 1.1.1.1

                !!!!!

                Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

                 

                msft-4-1002-b#ping vrf Twofish

                Protocol [ip]:

                Target IP address: 1.1.1.1

                Repeat count [5]:

                Datagram size [100]:

                Timeout in seconds [2]:

                Extended commands [n]: y

                Ingress ping [n]:

                Source address or interface: 2.2.2.2

                Type of service [0]:

                Set DF bit in IP header? [no]:

                Validate reply data? [no]:

                Data pattern [0x0000ABCD]:

                Loose, Strict, Record, Timestamp, Verbose[none]:

                Sweep range of sizes [n]:

                Type escape sequence to abort.

                Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:

                Packet sent with a source address of 2.2.2.2

                !!!!!

                Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

                 

                 

                OK, now were getting somewhere!  What do these outputs tell us? 

                 

                What they say is that the WAN_router has not received all the routes from the old_network, while the old_network has gotten all the routes from WAN_router.  Specifically, we see that the WAN_router hasn’t received 172.30.1.4/31 from its ACI peer, Leaf 102.  Passing routes from an L3Out border leaf to its non-ACI peer is controlled in the external EPG, by the subnets configured with the export route control flag.

                 

                Let’s take a look at what the ACI fabric is passing to the WAN_router:

                 

                 

                Screen Shot 2019-08-22 at 9.16.33 AM.png

                 

                See how this border leaf (Leaf 101 peering with the old_network) is advertising two subnets to its peer, where above, Leaf 102 was only advertising a single subnet?

                 

                Let’s go into the ACI GUI, fix the problem, and check our pings again. 

                 

                WAN_router#ping vrf Twofish 2.2.2.2

                Type escape sequence to abort.

                Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:

                !!!!!

                Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

                 

                old_network#ping vrf Twofish 1.1.1.1

                Type escape sequence to abort.

                Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:

                !!!!!

                Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

                The MoTS:

                • Understand what your tools tell you.
                • Understand what the external EPG flags do. 
                • See... networking fundamentals are still important!

                 

                If you answered this Troubleshoot Challenge...go on you!  If you mentioned something about external EPG flags and extended pings, I gave you credit.  A big thank you to Jerome Tissieres and Jacob for submitting answers.  For folks who didn't submit answers, I hope you enjoyed this troubleshoot challenge.

                 

                MM

                • 5. Re: The Adventures of Layer One Larry (#1):  I pinged what?
                  Jacob

                  Thanks for the challenge, Micheline! I look forward to the next one!

                  • 6. Re: The Adventures of Layer One Larry (#1):  I pinged what?
                    Jerome Tissieres

                    Thank you very much Micheline for this challenge and the very detailed answer. Very interesting!

                     

                    Jerome

                    • 7. Re: The Adventures of Layer One Larry (#1):  I pinged what?
                      Elvin Arias

                      Nice one, keep them coming!

                       

                      Elvin