12 Replies Latest reply: Jul 19, 2019 10:10 AM by alejo-VIRL Support RSS

    Issue with 1.6 upgrade

    bhartsfield

      So I just tried to do the 1.6 Upgrade.  Selected the full uprgade, option, went to maintenance mode and told it to start upgrade.  Upgrade button is flashing yellow. 

       

      However, it's been over an hour and I can't tell that anything is even happening.  All the log file outputs on the web page show blank.   I go into the CLI and under /var/logl/virl/logs none of the upgrade* log files have a current date on them.   Only log files updated today are std_server.log and uwm_server.log.    Looking at process list see nothing upgrade related in the list though I do have /usr/local/bin/virl_uwm_server run is pushing 90-100% CPU.

       

      Do I need to be concerned?  Should I just wait?   The lack of log file updates is what has me a little worried. 

        • 1. Re: Issue with 1.6 upgrade
          zx122982685

          Installation progress is indicated on this page.  Where are you on it. 

           

          • 2. Re: Issue with 1.6 upgrade
            Karlo Bobiles

            Hello bhartsfield,

             

            As WhiskeySixRadio stated, can you please take a screenshot of your UWM console? If you can expand "Progress" to see where in the upgrade process you are currently at, that would be greatly appreciated.

             

            Thank you,
            Karlo Bobiles

            Cisco Learning Network

            • 3. Re: Issue with 1.6 upgrade
              bhartsfield

              Screen Shot 2019-07-18 at 12.29.54 PM.pngScreen Shot 2019-07-18 at 12.29.10 PM.pngScreen Shot 2019-07-18 at 12.29.01 PM.pngScreen Shot 2019-07-18 at 12.28.42 PM.png

              • 4. Re: Issue with 1.6 upgrade
                bhartsfield

                Posted.  Doesn't look like it's doing anything.   All of the log windows look just like the 2 I posted

                • 5. Re: Issue with 1.6 upgrade
                  Karlo Bobiles

                  Hello bhartsfield,

                   

                  Thank you for the updated information. Looping in mirlos and alegalle for further investigation, please stand by.

                   

                  Regards,

                  Karlo Bobiles
                  Cisco Learning Network

                  • 6. Re: Issue with 1.6 upgrade
                    alejo-VIRL Support

                    bhartsfield close your browser and clear all cookies and cache for your VIRL Server IP address, then reboot the server via console and log back in.

                    Once the server has rebooted please collect the logs and post them in your reply.

                    • 7. Re: Issue with 1.6 upgrade
                      bhartsfield

                      After doing the reboot looks like it kicked it when the system came back up.  Appears to be running now. 

                      • 8. Re: Issue with 1.6 upgrade
                        bhartsfield

                        Upgrade ran and did reboot and now I'm getting a 503 service unavailable error.  I did catch this under the vinstall section while the upgrade was going on which might be related to the issue. 
                        ---------

                         

                          "cmd_|-docker_repository_key_|-apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-keys 9DC858229FC7DD38854AE2D88D81803C0EBFCD88_|-run": {

                            "comment": "Command \"apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-keys 9DC858229FC7DD38854AE2D88D81803C0EBFCD88\" run",

                            "name": "apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-keys 9DC858229FC7DD38854AE2D88D81803C0EBFCD88",

                            "start_time": "18:27:29.483437",

                            "result": false,

                            "duration": 122680.531,

                            "__run_num__": 5,

                            "__sls__": "base.docker",

                            "changes": {

                              "pid": 23553,

                              "retcode": 2,

                              "stderr": "gpg: requesting key 0EBFCD88 from hkp server ha.pool.sks-keyservers.net\ngpg: keyserver timed out\ngpg: keyserver receive failed: keyserver error",

                              "stdout": "Executing: /tmp/tmp.wmjd60UAGY/gpg.1.sh --keyserver\nhkp://ha.pool.sks-keyservers.net:80\n--recv-keys\n9DC858229FC7DD38854AE2D88D81803C0EBFCD88"

                         

                        ------------------------------------------

                         

                         

                        === State base.docker ===

                        Succeeded 6; failed 1

                        Duration: 235.42

                        ERROR: Some states failed in call "state.sls base.docker":

                        "cmd_|-docker_repository_key_|-apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-keys 9DC858229FC7DD38854AE2D88D81803C0EBFCD88_|-run" (Command "apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-keys 9DC858229FC7DD38854AE2D88D81803C0EBFCD88" run)

                        Upgrade failed on base.docker

                        • 9. Re: Issue with 1.6 upgrade
                          bhartsfield

                          Here is the log file with the failed message in it.

                          • 10. Re: Issue with 1.6 upgrade
                            bhartsfield

                            This does appear related to that docker failure.  in syslog server failed to start due to "no module docker".  Line 15 and 25 and if I type docker on the command line get told docker is not currently installed. 

                             

                             

                            Jul 18 18:43:17 virl virl_issue[9398]: virl_issue: The MAC OUI on eth0 is 00:50:56.
                            Jul 18 18:43:17 virl virl_issue[9398]: virl_issue: Determining the deployment platform.
                            Jul 18 18:43:18 virl virl_webmuxd[9403]: 2019-07-18 18:43:18+0000 [-] Log opened.
                            Jul 18 18:43:18 virl virl_webmuxd[9403]: 2019-07-18 18:43:18+0000 [-] Namespace(port=19403)
                            Jul 18 18:43:18 virl virl_webmuxd[9403]: 2019-07-18 18:43:18+0000 [-] Site starting on 19403
                            Jul 18 18:43:18 virl virl_webmuxd[9403]: 2019-07-18 18:43:18+0000 [-] Starting factory <twisted.web.server.Site instance at 0x7fb69b559f38>
                            Jul 18 18:43:18 virl virl_std_server[9386]: Traceback (most recent call last):
                            Jul 18 18:43:18 virl virl_std_server[9386]:   File "/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/bin/virl_std_server", line 24, in <module>
                            Jul 18 18:43:18 virl virl_std_server[9386]:    
                            Jul 18 18:43:18 virl virl_std_server[9386]:   File "/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/virl/std/management.py", line 42, in management
                            Jul 18 18:43:18 virl virl_std_server[9386]:   File "/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/virl/low_level/lxc.py", line 31, in lxc
                            Jul 18 18:43:18 virl virl_std_server[9386]:   File "/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/virl/low_level/implementation.py", line 43, in implementation
                            Jul 18 18:43:18 virl virl_std_server[9386]:   File "/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/virl/common/coreos.py", line 11, in coreos
                            Jul 18 18:43:18 virl virl_std_server[9386]:   File "/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/virl/fix/for_docker.py", line 1, in for_docker
                            Jul 18 18:43:18 virl virl_std_server[9386]: ImportError: No module named docker
                            Jul 18 18:43:18 virl kernel: [   72.269927] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
                            Jul 18 18:43:18 virl virl_uwm_server[9401]: Traceback (most recent call last):
                            Jul 18 18:43:18 virl virl_uwm_server[9401]:   File "/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/bin/virl_uwm_server", line 29, in <module>
                            Jul 18 18:43:18 virl virl_uwm_server[9401]:    
                            Jul 18 18:43:18 virl virl_uwm_server[9401]:   File "/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/virl/std/management.py", line 42, in management
                            Jul 18 18:43:18 virl virl_uwm_server[9401]:   File "/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/virl/low_level/lxc.py", line 31, in lxc
                            Jul 18 18:43:18 virl virl_uwm_server[9401]:   File "/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/virl/low_level/implementation.py", line 43, in implementation
                            Jul 18 18:43:18 virl virl_uwm_server[9401]:   File "/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/virl/common/coreos.py", line 11, in coreos
                            Jul 18 18:43:18 virl virl_uwm_server[9401]:   File "/var/jenkins/workspace/VIRL_CORE_build/test-virl-repo/virl/fix/for_docker.py", line 1, in for_docker
                            Jul 18 18:43:18 virl virl_uwm_server[9401]: ImportError: No module named docker
                            Jul 18 18:43:18 virl systemd[1]: virl-std.service: Main process exited, code=exited, status=1/FAILURE
                            Jul 18 18:43:18 virl systemd[1]: Started "Configure IP Tables for VIRL.".
                            Jul 18 18:43:18 virl systemd[1]: virl-uwm.service: Main process exited, code=exited, status=1/FAILURE
                            Jul 18 18:43:18 virl systemd[1]: Stopping VIRL webmuxd...
                            Jul 18 18:43:18 virl virl_webmuxd[9403]: 2019-07-18 18:43:18+0000 [-] Received SIGTERM, shutting down.
                            Jul 18 18:43:18 virl virl_webmuxd[9403]: 2019-07-18 18:43:18+0000 [twisted.web.server.Site] (TCP Port 19403 Closed)
                            Jul 18 18:43:18 virl virl_webmuxd[9403]: 2019-07-18 18:43:18+0000 [-] Stopping factory <twisted.web.server.Site instance at 0x7fb69b559f38>
                            Jul 18 18:43:18 virl virl_webmuxd[9403]: 2019-07-18 18:43:18+0000 [-] Main loop terminated.
                            Jul 18 18:43:19 virl neutron-linuxbridge-agent[11197]: Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
                            Jul 18 18:43:19 virl neutron-linuxbridge-cleanup[10180]: Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the futur
                            
                            
                            • 11. Re: Issue with 1.6 upgrade
                              Ben

                              Experiencing same exact failure in my upgrade... hoping the group will have a fix.

                              • 12. Re: Issue with 1.6 upgrade
                                alejo-VIRL Support

                                Please see: VIRL 1.6.65 Release Notes Start Here section.