9 Replies Latest reply: Sep 4, 2019 3:27 AM by Fabio - Data Center Engineer RSS

    Help!! vPC not working.

    adrianfer

      Good afternoon,

       

      I can not flip my vpc traffic when reloading my primary vpc switch. Here is the config:

      LAB:

      vPC case.png

       

      NXOS1 config:

       

      NXOS1# sh run vpc

       

      !Command: show running-config vpc

      !Time: Tue Aug  6 21:48:45 2019

       

      version 7.0(3)I7(4)

      feature vpc

       

      vpc domain 1

        peer-switch

        role priority 20

        peer-keepalive destination 10.1.2.2 source 10.1.2.1 vrf default

        peer-gateway

        auto-recovery

        ip arp synchronize

       

      interface port-channel20

        vpc peer-link

       

      interface port-channel30

        vpc 30

       

      interface port-channel40

        vpc 40

       

       

       

      NXOS1# sh vpc brief

      Legend:

                      (*) - local vPC is down, forwarding via vPC peer-link

       

      vPC domain id                     : 1

      Peer status                       : peer adjacency formed ok

      vPC keep-alive status             : peer is alive

      Configuration consistency status  : success

      Per-vlan consistency status       : success

      Type-2 consistency status         : success

      vPC role                          : primary

      Number of vPCs configured         : 2

      Peer Gateway                      : Enabled

      Dual-active excluded VLANs        : -

      Graceful Consistency Check        : Enabled

      Auto-recovery status              : Enabled, timer is off.(timeout = 240s)

      Delay-restore status              : Timer is off.(timeout = 30s)

      Delay-restore SVI status          : Timer is off.(timeout = 10s)

      Operational Layer3 Peer-router    : Disabled

       

      vPC Peer-link status

      ---------------------------------------------------------------------

      id    Port   Status Active vlans

      --    ----   ------ -------------------------------------------------

      1     Po20   up     1-2

       

       

      vPC status

      ----------------------------------------------------------------------------

      Id    Port          Status Consistency Reason                Active vlans

      --    ------------  ------ ----------- ------                ---------------

      30    Po30          up     success     success               1-2

       

       

       

      40    Po40          up     success     success               1-2

      NXOS1# sh vpc role

       

      vPC Role status

      ----------------------------------------------------

      vPC role                        : primary

      Dual Active Detection Status    : 0

      vPC system-mac                  : 00:23:04:ee:be:01

      vPC system-priority             : 32667

      vPC local system-mac            : 50:00:00:01:00:07

      vPC local role-priority         : 20

      vPC local config role-priority  : 20

      vPC peer system-mac             : 50:00:00:02:00:07

      vPC peer role-priority          : 30

      vPC peer config role-priority   : 30

       

      NXOS1# sh spanning-tree summary

      Switch is in rapid-pvst mode

      Root bridge for: VLAN0001-VLAN0002

      L2 Gateway STP                           is disabled

      Port Type Default                        is disable

      Edge Port [PortFast] BPDU Guard Default  is disabled

      Edge Port [PortFast] BPDU Filter Default is disabled

      Bridge Assurance                         is enabled

      Loopguard Default                        is disabled

      Pathcost method used                     is short

      vPC peer-switch                          is enabled (operational)

      STP-Lite                                 is disabled

       

      Name                   Blocking Listening Learning Forwarding STP Active

      ---------------------- -------- --------- -------- ---------- ----------

      VLAN0001                     0         0        0          4          4

      VLAN0002                     0         0        0          4          4

      ---------------------- -------- --------- -------- ---------- ----------

      2 vlans                      0         0        0          8          8

       

      NXOS2 config:

       

      NXOS2# sh run vpc

       

      !Command: show running-config vpc

      !Time: Tue Aug  6 21:49:46 2019

       

      version 7.0(3)I7(4)

      feature vpc

       

      vpc domain 1

        peer-switch

        role priority 30

        peer-keepalive destination 10.1.2.1 source 10.1.2.2 vrf default

        peer-gateway

        auto-recovery

        ip arp synchronize

       

      interface port-channel20

        vpc peer-link

       

      interface port-channel30

        vpc 30

       

      interface port-channel40

        vpc 40

       

       

       

      NXOS2# sh vpc brief

      Legend:

                      (*) - local vPC is down, forwarding via vPC peer-link

       

      vPC domain id                     : 1

      Peer status                       : peer adjacency formed ok

      vPC keep-alive status             : peer is alive

      Configuration consistency status  : success

      Per-vlan consistency status       : success

      Type-2 consistency status         : success

      vPC role                          : secondary

      Number of vPCs configured         : 2

      Peer Gateway                      : Enabled

      Dual-active excluded VLANs        : -

      Graceful Consistency Check        : Enabled

      Auto-recovery status              : Enabled, timer is off.(timeout = 240s)

      Delay-restore status              : Timer is off.(timeout = 30s)

      Delay-restore SVI status          : Timer is off.(timeout = 10s)

      Operational Layer3 Peer-router    : Disabled

       

      vPC Peer-link status

      ---------------------------------------------------------------------

      id    Port   Status Active vlans

      --    ----   ------ -------------------------------------------------

      1     Po20   up     1-2

       

       

      vPC status

      ----------------------------------------------------------------------------

      Id    Port          Status Consistency Reason                Active vlans

      --    ------------  ------ ----------- ------                ---------------

      30    Po30          up     success     success               1-2

       

       

       

      40    Po40          up     success     success               1-2

       

      NXOS2# sh vpc role

       

      vPC Role status

      ----------------------------------------------------

      vPC role                        : secondary

      Dual Active Detection Status    : 0

      vPC system-mac                  : 00:23:04:ee:be:01

      vPC system-priority             : 32667

      vPC local system-mac            : 50:00:00:02:00:07

      vPC local role-priority         : 30

      vPC local config role-priority  : 30

      vPC peer system-mac             : 50:00:00:01:00:07

      vPC peer role-priority          : 20

      vPC peer config role-priority   : 20

      NXOS2#

      NXOS2#

      NXOS2# sh spanning-tree summary

      Switch is in rapid-pvst mode

      Root bridge for: VLAN0001-VLAN0002

      L2 Gateway STP                           is disabled

      Port Type Default                        is disable

      Edge Port [PortFast] BPDU Guard Default  is disabled

      Edge Port [PortFast] BPDU Filter Default is disabled

      Bridge Assurance                         is enabled

      Loopguard Default                        is disabled

      Pathcost method used                     is short

      vPC peer-switch                          is enabled (operational)

      STP-Lite                                 is disabled

       

      Name                   Blocking Listening Learning Forwarding STP Active

      ---------------------- -------- --------- -------- ---------- ----------

      VLAN0001                     0         0        0          4          4

      VLAN0002                     0         0        0          4          4

      ---------------------- -------- --------- -------- ---------- ----------

      2 vlans                      0         0        0          8          8

      NXOS2#

       

      SITUATION:

       

      Step 1 - R3 pings R8:

       

      R3#ping 10.1.1.4 repeat 100000 source 10.1.1.1

      Type escape sequence to abort.

      Sending 100000, 100-byte ICMP Echos to 10.1.1.4, timeout is 2 seconds:

      Packet sent with a source address of 10.1.1.1

      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

       

      step 2 - I reload NXOS1:

       

      NXOS1#

      NXOS1# reload

      This command will reboot the system. (y/n)?  [n] y

      2019 Aug  6 21:39:14 NXOS1 %$ VDC-1 %$ %PLATFORM-2-PFM_SYSTEM_RESET: Manual system restart from Command Line Interface

       

      step 3 - I loose pings on R3 and NXOS2 lose connection with NXOS1 :

       

      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

      !................

       

      NXOS2# sh vpc brief

      Legend:

                      (*) - local vPC is down, forwarding via vPC peer-link

       

      vPC domain id                     : 1

      Peer status                       : peer link is down

      vPC keep-alive status             : Suspended (Destination IP not reachable)

      Configuration consistency status  : success

      Per-vlan consistency status       : success

      Type-2 consistency status         : success

      vPC role                          : secondary, operational primary

      Number of vPCs configured         : 2

      Peer Gateway                      : Enabled

      Dual-active excluded VLANs        : -

      Graceful Consistency Check        : Enabled

      Auto-recovery status              : Enabled, timer is off.(timeout = 240s)

      Delay-restore status              : Timer is off.(timeout = 30s)

      Delay-restore SVI status          : Timer is off.(timeout = 10s)

      Operational Layer3 Peer-router    : Disabled

       

      vPC Peer-link status

      ---------------------------------------------------------------------

      id    Port   Status Active vlans

      --    ----   ------ -------------------------------------------------

      1     Po20   down   -

       

       

      vPC status

      ----------------------------------------------------------------------------

      Id    Port          Status Consistency Reason                Active vlans

      --    ------------  ------ ----------- ------                ---------------

      30    Po30          up     success     Type checks were      1-2

       

                                             bypassed for the vPC

       

      40    Po40          up     success     Type checks were      1-2

       

                                             bypassed for the vPC

       

       

       

      step 4 - Pings only come back almost 4 minutes after the reload, some seconds after the keep-alive link and peer link go up.

       

      What I am doing wrong? This needs to be transparent.

        • 1. Re: Help!! vPC not working.
          Micheline

          Hello Adrianfer--it looks like your failure scenario is one in which the peer-link and the peer-keepalive are failing.

           

          In the event that the peer-link fails, the secondary pings the primary on the peer-keepalive.  If the ping returns, then the secondary knows that the primary is up, and the secondary will shut down its member ports and SVIs.  BUT, if the ping fails, the secondary will think it's all on its own now, and keep its links up.

           

          But, that does not seem to be happening here.  I notice that you got this message:

           

          2019 Aug  6 21:39:14 NXOS1 %$ VDC-1 %$ %PLATFORM-2-PFM_SYSTEM_RESET: Manual system restart from Command Line Interface (Emphasis added)

          I see that you're on a Nexus 7K, with VDCs configured.  In order to reload a single VDC on a 7K that's been split up into mulitple VDCs, you need to use the command reload VDC.  The reload command has the effect of popping every VDC.  So what you probably did is reload both peer switches, and your pings came back up only after both VDCs came back up and the 4 minute VPC timer expired.  But in the meantime, all of your downstream endpoints lost their connectivity.

           

          Interested to hear if my theory is right . MM

          • 2. Re: Help!! vPC not working.
            adrianfer

            Hi Micheline,

            Thanks for your reply. As you can see on the diagram NXOS1 and NXOS2 are two physical switches, I did not configure any VDC on each swtch, and the VDC1 you see going down is just the default VDC of NXOS1.

            • 3. Re: Help!! vPC not working.
              Jacob

              Hi Adrianfer,

               

              I was able to replicate this using physical 7Ks and 5Ks, and failover worked correctly on my end.

               

              Quick question - which version of NXOSv are you using? I've had trouble with vPC in the past on some images.

               

              Thanks!

              • 4. Re: Help!! vPC not working.
                adrianfer

                Hi Jacob,

                 

                Thanks for your reply both Nexus are using the following:

                 

                 

                Nexus 9000v is a demo version of the Nexus Operating System

                 

                Software

                  BIOS: version

                  NXOS: version 7.0(3)I7(4)

                  BIOS compile time:

                  NXOS image file is: bootflash:///nxos.7.0.3.I7.4.bin

                  NXOS compile time:  6/14/2018 2:00:00 [06/14/2018 09:49:04]

                 

                 

                over EVE-NG.

                 

                I saw a post over internet using vpc with nxos.7.0.3.I7.2 over EVE-NG, I will try it, what NWOS version you did use?

                • 5. Re: Help!! vPC not working.
                  adrianfer

                  I meant NXOS not NWOS

                  • 6. Re: Help!! vPC not working.
                    Jerome Tissieres

                    Hi Adrianfer,


                    I had similar problems with EVE-NG no later than yesterday: two Nexus 9k in vPC and when I reloaded the primary the traffic did not switched totally to the 2nd path. I discovered that when you reload a node, the link on the other side - for example e0/0 of SW1 on your case if your reload NXOS1 - stayed up.


                    Did you check the status of e0/0 on SW1 when you reload NXOS1? Does it goes down?


                    I hope this may help.

                    Best Rgds,

                    Jerome

                    • 7. Re: Help!! vPC not working.
                      adrianfer

                      Hi Jerome,

                       

                      Thanks for your reply. I tried it your way with no luck, but it let me dig a bit more on the issue, here is what I got:

                       

                      Some introduction:

                       

                      My test is a ping from R3 to R8 ip 10.1.1.4, so here is the destination ip's MAC:

                       

                      R3#sh ip arp

                      Protocol  Address          Age (min)  Hardware Addr   Type   Interface

                      Internet  10.1.1.1                -   aabb.cc00.3000  ARPA   Ethernet0/0

                      Internet  10.1.1.2               43   aabb.cc00.4000  ARPA   Ethernet0/0

                      Internet  10.1.1.3               43   aabb.cc00.6000  ARPA   Ethernet0/0

                      Internet  10.1.1.4               43   aabb.cc00.8000  ARPA   Ethernet0/0

                       

                       

                      R8#sh arp

                      Protocol  Address          Age (min)  Hardware Addr   Type   Interface

                      Internet  10.1.1.1               11   aabb.cc00.3000  ARPA   Ethernet0/0

                      Internet  10.1.1.2               11   aabb.cc00.4000  ARPA   Ethernet0/0

                      Internet  10.1.1.3               11   aabb.cc00.6000  ARPA   Ethernet0/0

                      Internet  10.1.1.4                -   aabb.cc00.8000  ARPA   Ethernet0/0

                       

                      Do SW1 see the MAC?


                      SW1#sh mac address-table

                                Mac Address Table

                      -------------------------------------------

                       

                      Vlan    Mac Address       Type        Ports

                      ----    -----------       --------    -----

                         2    0000.0c07.ac01    DYNAMIC     Po10

                         2    aabb.cc00.3000    DYNAMIC     Et0/2

                         2    aabb.cc00.4000    DYNAMIC     Po10

                         2    aabb.cc00.6000    DYNAMIC     Po10

                         2    aabb.cc00.8000    DYNAMIC     Po10

                       

                      Yes, it knows about the MAC and it is reachable through the right port Po10.

                       

                      Running the test:

                       

                      I start the ping between R3 and R8.

                      Once I reload NXOS1, e0/0 on SW1 goes down, but Port-Channel stay up:

                      SW1#sh ip int b

                      Interface              IP-Address      OK? Method Status                Protocol

                      Ethernet0/0            unassigned      YES unset  up                    down

                      Ethernet0/1            unassigned      YES unset  up                    up

                      Ethernet0/2            unassigned      YES unset  up                    up

                      Ethernet0/3            unassigned      YES unset  up                    up

                      Port-channel10         unassigned      YES unset  up                    up

                       

                      SW1#sh etherchannel summary

                      Flags:  D - down        P - bundled in port-channel

                              I - stand-alone s - suspended

                              H - Hot-standby (LACP only)

                              R - Layer3      S - Layer2

                              U - in use      N - not in use, no aggregation

                              f - failed to allocate aggregator

                       

                              M - not in use, minimum links not met

                              m - not in use, port not aggregated due to minimum links not met

                              u - unsuitable for bundling

                              w - waiting to be aggregated

                              d - default port

                       

                              A - formed by Auto LAG

                       

                       

                      Number of channel-groups in use: 1

                      Number of aggregators:           1

                       

                      Group  Port-channel  Protocol    Ports

                      ------+-------------+-----------+-----------------------------------------------

                      10     Po10(SU)        LACP      Et0/0(s)    Et0/1(P)

                       

                      Everything is fine here, but when I look at SW1 MAC address table I got the following:

                       

                      SW1#sh mac address-table

                                Mac Address Table

                      -------------------------------------------

                       

                      Vlan    Mac Address       Type        Ports

                      ----    -----------       --------    -----

                         2    0000.0c07.ac01    DYNAMIC     Et0/2

                         2    aabb.cc00.3000    DYNAMIC     Et0/2

                         2    aabb.cc00.4000    DYNAMIC     Po10

                      Total Mac Addresses for this criterion: 3

                       

                      aabb.cc00.8000 is not there any more. Obviously the is a problem with the ARP propagation between SW1 ans NXOS2. At this point I think I will open a case at EVE-NG.

                      • 8. Re: Help!! vPC not working.
                        Micheline

                        Please let us know how this resolved.  I'm very interested in a solution.  Thank you, MM

                        • 9. Re: Help!! vPC not working.
                          Fabio - Data Center Engineer

                          Hi Adrian,

                           

                          the vPC configuration seems to be ok; did you check the SW02 configuration and the mac-address table during the reboot process of the nexus01?

                           

                           

                          Regards,

                          Fabio