7 Replies Latest reply: Dec 11, 2019 6:36 PM by Ralph RSS

    Cisco VM (Cisco IOS XRv 9000) deployment time

    Thomas

      I am working on launching Cisco VMs (Cisco IOS XRv 9000) programmatically on OpenStack.

       

      During our VM deployment process, we have noticed that the Cisco VM takes close to 30 mins and sometimes 45 mins to be accessible over ssh after it has been deployed. We are using a sizably big instance – 65536 MB RAM, 128 GB disk and 32 VCPUS.

       

      1. Is it expected for the VM’s to take this long to come up?
      2. Is there anything that can be done to speed up the process?
      3. What is the limit to the number of interfaces on a iosxrv9k VM?
        • 1. Re: Cisco VM (Cisco IOS XRv 9000) deployment time
          alejo-VIRL Support

          The IOS-XRv nodes are very large in resource intensive virtual machines. They will take longer to boot than other devices, but 30 minutes for any node in VIRL is excessive to say the least. If your simulation has a lot of nodes, try staggering the start of the nodes. From VMM, select a group of nodes in the simulation and check the tick box "Do not start at simulation launch". Once the first set of nodes have started, you can then start the rest. This has to do with heavy IO at time of simulation launch. It is also a good idea to look at your VIRL host's disk and CPU performance. Even with a high resource server, the heavy load created at simulation launch can create a resource contention causing some nodes to fail to start or receive their start-up configuration.

           

          IOS-XRv 9k are capped at 16 interfaces.

          • 2. Re: Cisco VM (Cisco IOS XRv 9000) deployment time
            Thomas

            In order to triangulate the cause for the slow deployment, I have been launching one VM at a time. It still takes 30-45 min to launch a single VM.

             

            It is important to note that I am deploying the VM on OpenStack using Terraform. The creation of OpenStack resources - networks, ports and instance takes less than 30 seconds. The rest of the 30-45 min wait time is for the iosxrv9k VM to come up and to have operational ssh on the management port.

             

            Any other ideas for what could be causing the long wait?

            • 3. Re: Cisco VM (Cisco IOS XRv 9000) deployment time
              Thomas

              Here are some observations on deployment times from launching other kinds of instances on the same OpenStack flavor:

              • I launched an Ubuntu 16.04 on the exact same flavor of OpenStack used for the cisco VM and the Ubuntu instance was deployed and reachable in under a minute. I used Terraform to deploy this instance as well.
              • As this is a multi-vendor effort, I launched a VM by a different vendor on a smaller instance (4 vCPUs, 8 GB RAM and 20 GB disk) which came up in 2 minutes.

               

              Could it be that the Cisco VM performs repeated restarts and stabilizes after 30 min? Typically how long does the VM take to come up?

              • 4. Re: Cisco VM (Cisco IOS XRv 9000) deployment time
                Thomas

                I've attached a text file containing the console logs from this afternoon's VM deployment to this post.

                • 5. Re: Cisco VM (Cisco IOS XRv 9000) deployment time
                  Thomas

                  I have some interesting results.

                   

                  I conducted two tests where I deployed 2 Cisco iosxr9k VMs with hostnames R101 and R102 at a time. Once the VMs had ssh access with the credentials specified in the baked-in base config, I performed a soft reboot on the instances. The soft reboot performs a graceful shutdown and restart of the instance and is equivalent to stopping and starting the instance. I timed both the initial time to successful ssh login after deployment and time to successful ssh login post soft reboot.

                   

                  Test1:

                  Time taken to boot with ssh access after initial deployment:

                  R101 – 26 min

                  R102 – 22 min

                   

                  Time taken to boot up with ssh access after soft reboot:

                  R101 – 11 min

                  R102 – 10 min

                   

                  Test2:

                  Time taken to boot with ssh access after initial deployment:

                  R101 – 25 min

                  R102 – 22 min

                   

                  Time taken to boot up with ssh access after soft reboot:

                  R101 – 10 min

                  R102 – 10 min

                   

                  The deployment times were much shorter during these tests. I’m unsure as to what contributed to this hastening but it is a good development. Our OpenStack region/environment has been stable during these tests and previous tests.

                  • 6. Re: Cisco VM (Cisco IOS XRv 9000) deployment time
                    alejo-VIRL Support

                    The initial deployment would always take the longest. This is because the virtual machine and all network connections are being defined and orchestrated by OpenStack. I/O is at its highest during initial deployment of any simulation. Once the simulation has been orchestrated and started, a reboot of any virtual machine (node) would not require any input from OpenStack and thus much quicker.

                     

                    I forgot to mention, have you enabled "Store VM drives in RAM" setting under VIRL Server > System Configuration > Hardware ? Enabling this feature on servers with 16GB+ of memory greatly improves performance.

                     

                    Hope this helps.

                    • 7. Re: Cisco VM (Cisco IOS XRv 9000) deployment time
                      Ralph

                      the issue here is that the IOS XR 9000v instances, when started for the first time, do a lot of internal initialization and installation of packages etc.

                       

                      - re-configuring boot loader to accommodate for current CPU settings (including a reboot)

                      - expanding and installing packages

                      - creating and initializing internal VMs and containers representing RPs and LCs

                      - and a lot more.

                       

                      This all takes a lot of time and also is quite some burden on I/O and CPU. As you observed, subsequent boots are much faster. This is because those initial one-time things can be skipped.