13 Replies Latest reply: Nov 6, 2018 12:18 AM by Federico RSS

    Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)

    Darren

      Good evening all,

       

      I love Cisco and I really love VIRL as it does all that Packet Tracer and GNS3 can't.  A couple of months back my VIRL updated and since then when I create a design/simulation and try to start it the device icons turn green but I am unable to console into anything.  It brings up the console window however I don't see the boot up messages or the prompt.

       

      As far as I know nothing else has changed on the PC that I run it on.  I have tried a fresh install of the VM and Maestro but the problem remains the same.

       

      I would love to get it working again !

       

       

      Note:  I have attached a zip file containing srv_validate UWM logs and ProblemReport.

       

      Thankyou very much again

       

      Daz

        • 1. Re: Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)
          alejo-VIRL Support

          Hello Darren, thank you for collecting and attaching the logs to the post. Based on the logs provided it appears that your VIRL server lost connectivity on  2018-05-18 13:03... and has not recovered. The later log entries show no route from your local machine to the VIRL PE virtual machine.

           

          !ENTRY com.cisco.vmmaestro 4 0 2018-05-18 16:30:08.792

          !MESSAGE Error launching simulation.

          !STACK 0

          javax.ws.rs.ProcessingException: java.net.NoRouteToHostException: No route to host: connect at org.glassfish.jersey.client.internal.HttpUrlConnector$3.run(HttpUrlConnector.java:299

           

           

          Make sure that the IP address of your VIRL PE server has not changed if set to DHCP.

          Keep us posted.

          • 2. Re: Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)
            Darren

            Alejo.

             

            Thankyou for taking the time to reply to me.

             

            I have seen this message however I am slightly confused.  My VIRL VM server is using NAT (VMWare) and has an IP address 192.168.106.128/24.  This address is static NAT and has been configured in VM Maestro and has checked out ok.

             

            I have enclosed 2 images.  1 which shows VM Maestro 'seemingly happy' with the IP of the VIRL server, and a second of the full desktop showing what happens now when a terminal / console window attempts to open.

             

            Basically if i telnet to console the router will start to display the bootup details after approx 5-10 minutes however it stops as soon it starts to show the copyright text. (as shown in the attached imaged called desktop).

             

            Like i said, i love VIRL but as far as I know I have not caused this issue.

             

            Thankyou for your replies.

             

            Dazdesktop capture.jpg

             

             

             

             

             

             

            Maestro config.jpg

            • 3. Re: Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)
              Darren

              I have logged in to the VM and have run the 'Systems Operation Check'

               

              I have noticed that within the 'Sumulation Tests' there is the 'Simulation Launch' which fails the final stage with the following:

               

              Simulation fail.png

              • 4. Re: Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)
                alejo-VIRL Support

                Thanks Darren,

                Please run the following commands and post output:

                By default the system should be using e1000 driver.

                lspci -v | grep Ether

                lspci -k

                 

                Do you see a large number of "Link Down" entries?

                less /var/log/syslog | grep NIC

                 

                Next...

                Close all simulations.

                Start a new simulation (no more than 4 nodes) and wait for the simulation to complete or error

                Collect a new set of logs from VMM and UWM while the failed simulation is still running and attach to response.

                • 5. Re: Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)
                  Darren

                  Thankyou very much Alejo,

                   

                  Please find the results to your commands below.  I have also attached the configs (and more) as requested.

                  lspci -v | grep Ether

                   

                  02:00.0 Ethernet controller: Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) (rev 01)

                          DeviceName: Ethernet0

                   

                  lspci -k

                   

                  virl@virl:~$ lspci -k

                  00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 01)

                  Subsystem: VMware Virtual Machine Chipset

                  Kernel driver in use: agpgart-intel

                  00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 01)

                  Kernel modules: shpchp

                  00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 08)

                  Subsystem: VMware Virtual Machine Chipset

                  00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)

                  Subsystem: VMware Virtual Machine Chipset

                  Kernel driver in use: ata_piix

                  Kernel modules: pata_acpi

                  00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)

                  Subsystem: VMware Virtual Machine Chipset

                  Kernel modules: i2c_piix4

                  00:07.7 System peripheral: VMware Virtual Machine Communication Interface (rev 10)

                  Subsystem: VMware Virtual Machine Communication Interface

                  Kernel driver in use: vmw_vmci

                  Kernel modules: vmw_vmci

                  00:0f.0 VGA compatible controller: VMware SVGA II Adapter

                  Subsystem: VMware SVGA II Adapter

                  Kernel driver in use: vmwgfx

                  Kernel modules: vmwgfx

                  00:10.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 01)

                  Subsystem: VMware LSI Logic Parallel SCSI Controller

                  Kernel driver in use: mptspi

                  Kernel modules: mptspi

                  00:11.0 PCI bridge: VMware PCI bridge (rev 02)

                  00:15.0 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:15.1 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:15.2 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:15.3 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:15.4 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:15.5 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:15.6 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:15.7 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:16.0 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:16.1 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:16.2 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:16.3 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:16.4 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:16.5 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:16.6 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:16.7 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:17.0 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:17.1 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:17.2 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:17.3 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:17.4 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:17.5 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:17.6 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:17.7 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:18.0 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:18.1 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:18.2 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:18.3 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:18.4 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:18.5 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport mymgmgmgmg

                  00:18.6 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  00:18.7 PCI bridge: VMware PCI Express Root Port (rev 01)

                  Kernel driver in use: pcieport

                  Kernel modules: shpchp

                  02:00.0 Ethernet controller: Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) (rev 01)

                  DeviceName: Ethernet0

                  Subsystem: VMware PRO/1000 MT Single Port Adapter

                  Kernel driver in use: e1000

                  Kernel modules: e1000

                   

                  virl@virl:~$ less /var/log/syslog | grep NIC

                  Jun 12 17:24:21 virl kernel: [  12.531683] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

                  Jun 12 20:06:06 virl kernel: [ 9717.176286] e1000: eth0 NIC Link is Down

                  Jun 12 20:06:18 virl kernel: [ 9729.202061] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

                  Jun 12 20:07:44 virl kernel: [ 9815.375766] e1000: eth0 NIC Link is Down

                  Jun 12 20:07:48 virl kernel: [ 9819.384931] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

                  Jun 12 20:07:50 virl kernel: [ 9821.391792] e1000: eth0 NIC Link is Down

                  Jun 12 20:07:56 virl kernel: [ 9827.404992] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

                  Jun 12 20:08:04 virl kernel: [ 9835.423853] e1000: eth0 NIC Link is Down

                  Jun 12 20:08:08 virl kernel: [ 9839.432997] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

                  Jun 13 17:07:58 virl kernel: [17474.514585] e1000: eth0 NIC Link is Down

                  Jun 13 17:08:06 virl kernel: [17482.540036] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

                  Jun 13 20:07:15 virl kernel: [28228.861418] e1000: eth0 NIC Link is Down

                  Jun 13 20:07:23 virl kernel: [28236.895568] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

                  Jun 13 20:08:19 virl kernel: [28293.070047] e1000: eth0 NIC Link is Down

                  Jun 13 20:08:27 virl kernel: [28301.087509] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

                  Jun 13 20:08:29 virl kernel: [28303.093946] e1000: eth0 NIC Link is Down

                  Jun 13 20:08:37 virl kernel: [28311.111146] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

                  virl@virl:~$

                  • 6. Re: Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)
                    alejo-VIRL Support

                    change the adapter type from E1000 to vmxnet3. Take a look at this link for assistance: https://kb.vmware.com/s/article/2033748

                    Make sure the virtual machine has been cleanly shutdown prior to editing the file.

                    • 7. Re: Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)
                      Darren

                      lxpci -k

                       

                      03:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)

                              DeviceName: Ethernet0

                              Subsystem: VMware VMXNET3 Ethernet Controller

                              Kernel driver in use: vmxnet3

                              Kernel modules: vmxnet3

                       

                      The issue remains the same I am afraid.

                      • 8. Re: Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)
                        Darren

                        Can anyone else add any more meat to the bone on this?  The issue remains the same. 

                         

                        I certainly don't feel that I have used $200 or VIRL yet.  It looks like I may never.

                        • 9. Re: Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)
                          Karlo Bobiles

                          Hello Darren,

                           

                          I do apologize for the delay, and have requested an update from alegalle. Please stand by while we review other options.

                           

                          Thank you,

                          Karlo Bobiles

                          Cisco Learning Network

                          • 10. Re: Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)
                            alejo-VIRL Support

                            Sorry Darren for the slow response.

                            Please do the following and attach any error logs or screenshots you feel may be helpful.

                             

                            1. Stop all simulations
                            2. Ensure Web services are all green
                            3. Create or use a simple two node simulation with at least ANK configuration on both nodes (keep it simple)
                            4. Start the simulation and wait for nodes to become ACTIVE
                            5. Connect to VIRL PE virtual machine via SSH and run the following commands:
                              1. lspci -k |grep -A4 '\bEther'
                              2. tail -f /var/log/syslog |grep NIC
                            6. Wait for the failure to occur and then collect the system logs and generate a problem report from VM Maestro while the simulation is still running
                            7. Collect the logs and output from console and attach to response

                             

                            Thank you.

                             

                            How to: Collect Logs For VIRL Troubleshooting

                            • 11. Re: Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)
                              Darren

                              Thankyou very much for getting back to me.

                               

                              Today the message was slightly different and said about CPU hog.

                               

                              I have uploaded a YouTube clip to show you how slow it is going. 

                               

                              Just to reiterate, VIRL is great, and used to work like a dream on my PC.  One day it just suddenly stopped.  I have tried fresh installs etc.  The only change that I have made since the start is renaming the NIC type as you mentioned in a previous post.

                               

                              Thankyou very much

                               

                              The video is unlisted as I don't want other people to view it and think VIRL is not stable.  https://youtu.be/q2aOY7o67b0

                              • 12. Re: Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)
                                alejo-VIRL Support

                                Thank you Darren for all of the information.

                                So the CPU hog message is a long known issue which is completely benign. Ignore those messages on IOSv platforms.

                                 

                                The video and logs unfortunately only show that the simulation has been started and is functioning as expected. The fact that we are not seeing the disconnects anymore is a great improvement from what we saw prior to the vNIC change. What remains is the responsiveness of the simulation nodes (IOSv in your simulation).

                                 

                                The logs show a couple of things which I would question. The most important is the version of VIRL CORE and other components

                                 

                                Component Version(s)

                                VIRL-RELEASE    1.5.145

                                VIRL-CORE      0.10.32.19

                                 

                                The expected version is:  0.10.35.35

                                 

                                If you have not rebooted the VIRL PE virtual machine since upgrade, reboot it and try again. Otherwise I would suggest to download the latest OVA from your Learning at Cisco account and deploy anew.

                                 

                                Please keep me posted.

                                • 13. Re: Cisco devices no longer boot within VIRL (Active-Unreachable with slow boot/no prompt)
                                  Federico

                                  Hi Darren,

                                  did you solve the issue at the end? I have exactly the same problem VIRL CPU Hog errors

                                  and I still not able to use any device (stuck during the boot process)

                                   

                                  Thank you.

                                   

                                  Regards,

                                  Federico.