T-Shoot: UWM admin page unavailable; error: "unable to connect"

    Summary

    This typically happens on new deployments or when modifications to hostname, IP address or physical interface settings needed to be made. If not careful, some services will fail to start because the needed connectivity has been interrupted. The VIRL server uses its default eth4 to communicate with Openstack services which run in the background. If this interface is modified or not properly assigned, services like UWM and Openstack will fail to connect leaving UWM inaccessible or unusable.

     

    Correcting UWM connectivity

    Wether this is a new or existing deployment the following steps still apply. Do not skip ahead unless instructed to do so, skipping ahead would likely result in unneeded steps and having to wait for commands to complete. One of the commands likely needed (rehost), can take in excess of 15 minutes to complete.

     

    Objective

    UWM service and Openstack must be able to communicate via eth4 on 172.16.10.250 IP address and port 19400. It is extremely likely that configuration is not complete or incorrect information has been entered in one or more of the following files:

    /etc/virl.ini

    /etc/network/interfaces

    /etc/hosts

    /etc/hostname

    First we will make sure that the VIRL server is not listening on port 19400, then inspect the files to ensure the expected information has been correctly entered. Syntax is important, and in Linux case does matter. A hostname of virl is not the same as a hostname of VIRL!

     

    Getting Started

    Connect to your VIRL server via SSH (recommended). Do not try to use the console of the virtual machine as it may present roadblocks which are nothing more than a distraction.

     

    ssh virl@<server_ip

     

    • Check if port 19400 is in a listening state. If it is, then UWM web page should be available and displaying errors similar to: Failed to load resource usage: resource not available. The server may be down

    netstat -lan |grep 1940*

    virl@virl:~$ netstat -lan |grep 1940*
    tcp 0 0 0.0.0.0:19401 0.0.0.0:* LISTEN
    tcp 0 0 0.0.0.0:19402 0.0.0.0:* LISTEN
    tcp 0 0 0.0.0.0:19403 0.0.0.0:* LISTEN
    tcp 0 0 0.0.0.0:19406 0.0.0.0:* LISTEN
    tcp 0 0 0.0.0.0:19407 0.0.0.0:* LISTEN
    tcp6 0 0 :::19401 :::* LISTEN
    tcp6 0 0 :::19402 :::* LISTEN

    • Check connectivity on eth4 with a simple ping against the Openstack IP address 172.16.10.250. The ping command may return a different value or an echo response, we are just looking for the interface status.

     

    virl@virl:~$ ping -I eth4 172.16.10.250

    connect: Network is unreachable

     

    • Check VIRL server network interface configuration
      • less /etc/network/interfaces
    auto eth5
    iface eth4 inet static
      address 172.16.10.250
      netmask 255.255.255.0
      mtu 1500
      post-up ip link set eth4 promisc on
    auto lo
    iface lo inet loopback
    auto lo:1
    iface lo:1 inet loopback
      address 127.0.1.1
      netmask 255.0.0.0
    auto eth1
    iface eth1 inet static
      address 172.16.1.254/24
      netmask 255.255.255.0
      post-up ip link set eth1 promisc on
    auto eth2
    iface eth2 inet static
      address 172.16.2.254/24
      netmask 255.255.255.0
      post-up ip link set eth2 promisc on
    auto eth3
    iface eth3 inet static
      address 172.16.3.254/24
      netmask 255.255.255.0
      post-up ip link set eth3 promisc on
    auto eth0
    iface eth0 inet dhcp
      dns-nameservers 8.8.8.8 8.8.4.4

     

    Note that in above output the interface name is not the expected eth4 but rather eth5. This is not necessarily the problem, but it is something that we need to note. The interface name, must be valid and must be consistant across all VIRL server configuration files.

     

     

    More detail about interface names

    The names of your VIRL server’s interfaces does not have to be labeled as eth x, but the label the interfaces must be consistent across the VIRL configuration. Deviation from the traditional eth interface label is more common with bare-metal VIRL deployments. There are several ways to change them but we will not discuss them here.
    If you are not sure what the name of your internal network interface should be, list the contents of /sys/class/net/ system directory. The interfaces listed there, are all of the known interfaces which are available to be configured. If the interface eth5 is the indeed the correct interface name, a matching directory must be present when listing the system directory.

     

    • List contents of /sys/class/net directory

      virl@virl:~$ ls /sys/class/net/
      eth0 eth2 eth4 ifb1 ifb11 ifb13 ifb15 ifb17 ifb19 ifb20 ifb22 ifb24 ifb26 ifb28 ifb3 ifb31 ifb5 ifb7 ifb9 virbr0
      eth1 eth3 ifb0 ifb10 ifb12 ifb14 ifb16 ifb18 ifb2 ifb21 ifb23 ifb25 ifb27 ifb29 ifb30 ifb4 ifb6 ifb8 lo virbr0-nic


      Note that eth5 is not listed


    • Check configuration of internal network interface in VIRL settings file

      grep -E '^internalnet_' /etc/virl.ini

      internalnet_controller_hostname: controller
      internalnet_controller_IP: 172.16.10.250
      internalnet_port: eth4
      internalnet_IP: 172.16.10.250
      internalnet_network: 172.16.10.0
      internalnet_netmask: 255.255.255.0
      internalnet_gateway: 172.16.10.1

       

      Notice that virl.ini is configured to use eth4 and not eth5 as shown by the interfaces file.

     

    Resolution

    In this example the problem is that the internal network does not have an interface to attach to. This is verified by listing /sys/class/net/ directory which, in our example did not show a directory for eth5 interface. To correct the problem, we need to change /etc/network/interfaces file to reflect the correct and intended interface.

     

    Update / Correct configuration file(s)

    To edit the file(s) we will use the command line editor ‘nano’. If you are not familiar on how to use ‘nano’, take a look at mcandre/cheatsheets nano on GitHub.

    Using Nano – Basic commnads

    Files

    • Ctrl–O Save file
    • Ctrl–X Close file

    Cut and Paste

    • ALT–A  Start marking text
    • CTRL–K Cut marked text or line
    • CTRL–U Paste text
    • Open /etc/network/interfaces file with Nano

      sudo nano /etc/network/interfaces

    • Use your cursor keys to navigate the file.

    • Locate eth5 and change it to eth4

    • Save the file and close the editor

    • Reboot your server

      sudo reboot now

    • Re-connect to your server via ssh and log in; nothing else at the moment!

      • default username / password: virl / VIRL
    • Give your server time to finish boot and services to start, then open your browser and connect to UWM


    YOU ARE NOT DONE! Continue with Finalizing Configuration

     

    Finalizing Configuration

    UWM should now be accessiable and you should be able to log in and finalize your deployment. Again, DO NOT skip ahead and be sure to read each of the following sections to ensure you have a solid VIRL server.

     

    Install your VIRL license

     

    Configure VIRL Server

    DO NOT continue if your license did not apply correctly!

    If you do not need to make changes to hostname, management IP or network, Openstack networks (i.e. Flat, Flat1, SNAT) or DNS servers; proceed to Apply Settings

    If you do need to make changes to hostname, management IP or network, Openstack networks (i.e. Flat, Flat1, SNAT) or DNS servers; continue with following steps.

    • Open UWM and log in with default username and password of  uwmadmin / password
    • Navigate to VIRL Server > System Configuration
      • Edit all applicable fields as required. When done, click on Apply Changes
      • In confirmation page ensure that vinstall rehost is listed under Changes Impact section. If vinstall rehost is not listed, cancel operation and proceed to Apply Settings
    • Apply Settings
      • Enable Maintenance Mode
      • Apply Changes
      • Reboot server when prompted
      • Complete changes and Disable Maintenance Mode


    All done! You now have a fully functional VIRL server!