T-Shoot: Failed to collect current salt contact status

     

     

    Verify License

     

    Verify your license information is correctly entered in UWM

    1. Navigate to VIRL Server > Salt Configuration and Status
    2. Click on Check status now (this resolves most errors)
      • If error(s) persists, click on Reset keys and ID
      • Proceed to "Reenter License Information"

     

    Reenter License information

    VIRL 1.3 and 1.5

     

    1. Click on Load Config File
    2. Navigate to your license ".pem" file; select it and click Open
    3. Set Cisco Salt Masters based on your geographical location
    4. Ensure Master sign public key is set to "eft.pub (Regular Use)"

    5. Click Reset to apply license


    If after resetting your license UWM returns a 'Not Connected' error to all configured Cisco Salt Masters, proceed to Check access on ports TCP 4505-4506 below. It is not uncommon for one or sometimes two Salt Servers to return 'Not Connected'. You only need to be able reach one of the four Cisco Salt Masters for authentication.

     

    TIP: Configure only one Salt Master to expedite troubleshooting and how quickly your VIRL server returns available software during updates.


    List of Cisco Salt Masters

    [UPDATED 04/04/2018]

     

    VIRL 1.2 and older (no longer supported)

    US Salt Servers

    EU Salt Servers

     

     

    VIRL 1.3

    US Salt Servers

    vsm-us-1.virl.info

    vsm-us-2.virl.info

    vsm-us-3.virl.info

    vsm-us-4.virl.info

     

    EU Salt Servers

    vsm-eu-1.virl.info

    vsm-eu-2.virl.info

    vsm-eu-3.virl.info

    vsm-eu-4.virl.info

     

    Asia Pacific Salt Servers

    vsm-ap-1.virl.info

    vsm-ap-2.virl.info

    vsm-ap-3.virl.info

    vsm-ap-4.virl.info

     

    VIRL 1.5 and later

    US Salt Servers

    vsm-us-51.virl.info

    vsm-us-52.virl.info

    vsm-us-53.virl.info

    vsm-us-54.virl.info

    EU Salt Servers

    vsm-eu-51.virl.info

    vsm-eu-52.virl.info

    vsm-eu-53.virl.info

    vsm-eu-54.virl.info

    Asia Pacific Salt Servers

    vsm-ap-51.virl.info

    vsm-ap-52.virl.info

    vsm-ap-53.virl.info

    vsm-ap-54.virl.info

     

    Troubleshooting License Errors and Connectivity

    Check Access on ports TCP 4505 and 4506

     

    Run the following command for each salt server:

    nc -zv [salt-server] 4505-4506

    Example:

    nc -zv vsm-us-51.virl.info 4505-4506

     

    OR:

     

    You can automate the port check and license validation using the VIRL Server Salt Connectivity Validation script. Make sure to read the description to understand the expected output generated by the script.

    If the "Testing Server" commands do not return "Success", you will need to investigate your network. It is likely that a firewall or home router is not allowing the connection to Cisco License servers on ports TCP 4505 and/or 4506. You will not be able to authenticate your server until this is resolved. The command that follows the port test will check your license validity. If the return is True, your license is valid and has been accepted. If the license has not been accepted, then an error is returned. Please collect all of the output so it can be used as supporting details if you choose to start a support thread.

     

    Manual Troubleshooting

     

    If you used the VIRL Server Salt Connectivity Validation script to collect your system's information, the steps below are not required for assistance. Running the commands below will only assist as additional information and will NOT correct any problem you may be experiencing with your license.

     

    Check License ID

     

    This command should return the license key ID you entered in Salt Domain and ID under Salt Configuration and Status.

    sudo salt-call --local grains.get id

     

    Check License Validation (Salt)

     

    If you have corrected the problem, this command will return True. If the command returns rejected, check the following:

    • Has your subscription expired?
    • Have you recently purchased a new license?
      • Check that your new license ID has not changed.
      • Ensure that you have entered the complete and correct PEM key. (VIRL 1.2 and lower)
    • Have you recently renewed your subscription?
      • Check that you were able to renew during grace period, otherwise you may have received a new license key.

    time sudo salt-call -l debug test.ping

     

    Check DNS settings

     

    If the previous salt-call command returns True, but the time (real) summary reports that the call took more than 5-10 seconds (ideally much lower than that), please check the DNS settings of the server.

    Check that the complete list of nameserver entries is as you'd expect from other machines in your network:

    cat /etc/resolv.conf

    If you suspect one of the nameservers, say 8.8.4.4, fails to resolve one of the salt masters in a timely manner, say vsm-eu-51.virl.info, test this with:

     

    time dig @8.8.4.4 A vsm-eu-51.virl.info

    The output should indicate that the server replied in a short time, and resolved the IP address of the host correctly. If not, consider using only salt master hosts that work properly for your network, or DNS servers which can resolve the salt masters properly. You may check the UWM System configuration for DNS server settings, as well as your DHCP server, if your primary (management) network uses DHCP.

     

    Compare your PEM key

     

    You can validate that the key text you have entered in UWM > Salt Configuration and Status, is indeed the exact text of your license file. Follow the steps below to compare your text and license key using a hashing algorithm.


    Step 1. Open a terminal connection to your VIRL server and run the following command:

    sudo sha512sum /etc/salt/pki/minion/minion.pem

    Make a note of the returned value so you can compare it to the value returned in Step 5.

     

    Step 2. Open your license file using a text editor like Notepad or Textedit. Do not use a word processor like MS Word or Apple Pages.

    Step 3. Type sha512sum << EOF in your terminal window and press enter so your prompt looks as shown:

    sudo sha512sum << EOF

    >

    Step 4. Copy all of the text in your license file and paste it into the terminal window.

    Step 5. Type EOF and press enter to calculate your sum. Your terminal content should look like this:

    sudo sha512sum << EOF

    > -----BEGIN RSA PRIVATE KEY-----

    > MIIEowIBAAKCAQEAvE2150xOkSg1dTI12GNOCZYC0vBeFD/phQn6kQ+ARrqhwKAA

    > dRh4BWIdi8+Ds9c1bw8ZpYeKlvm8M/Vr5RusT0Vmq4WBa4GknrQLyyRStdHrjHGG

    > RdCyQdWNlrPMv5I17NdMPz4+TVrmlFG8QQf0VmWomRvyZfTHKoLjtJy1X5wZrjSD

    > 1ApzacGIj1KRdqCo15PaYYJfrBqEuLLChkDlmUAUwYUgvPusxkby6HyMgPFgOAQj

    > 913b34J7vOH+ziqmp3JFAdftDlXEKHwRi/wQGK1ciPHpSkqFC5It1PfH7DKU0hXo

    > Jxa9StlnCpXtRcjmsu/4jAJHhWgY2a97/hjWwQIDAQABAoIBAFK+WxZeI5AJeSQc

    > giipylrP/kGuuchwJS7+qYogo59w1y8Ln+fUbrj+2jjbYOYfxS0eF4jzPGUHLDow

    > t86fEGJs/qaULcCN4UaTmzMSrP2UVssJ/5SiFhJzwpncPDZaCzQdjzWjJkFblkLK

    > f8ybsDRey20sKKHAo9v8VA7DZbey4CzOEsycoebSfn+xHCIvC6okTj2BrMQ45eE2

    > YhY/xEpSIgQ6dEGqdBTi7FOZv+uP69MOPPfNtTofv8xVBhMGIMnt0ukxhzxKoJgB

    > J3I1npbhmmjlK/jEoK2huDUr87/6mr+QH1Xsj5bRoORpNd2RKAIBOHcdRICw4NWH

    > G1+6a1kCgYEA9dbcVCkB2e8Fo5os2gS3BEmFbNAnHsSJ3BSI3etVx+nwT0pnHDkr

    > izOE9BJR01PX+Do9iURVvVHtI95u4TfoKmAjHGqSx+ype+yKrK+Kw41Qsk1PjV9B

    > RfY3foEKlf+6vqxGzEiAju3hMhO4FGu7eXwLagsQZdgBZTkQ49eNMiOj1b0fDzux

    > 60hPED4JV9vLQ4ygtgp0HN0+XjdLJIquhXPuYax42JEouAqTz08Xu4VM+5uOi2j4

    > PW9LNBfcWjo7bzglZ+KZfQKBgDhVT+fWt37BDxADw/AyOykQomKuzD2QP4ANcFRn

    > PedvUv5YPuMr17uI4sCr0gGqB0J1FkUNgNi6svHZoHdQgqp86IQVUbO567zPD511

    > mAXJcUBVTZwVXL9D/OtRK3Lm2TISQ18SseiEfa7DNiQbP3VazO4aVbZLtz8CYZKi

    > sKZZAoGBAM9ez92+CZC4UlcZCUHY99/qZ0/WRbjz8w/PiPksqpewmmi5wia3XzeC

    > gsPYhUuPMqupvquu0Wo445BR16nojwvPQeY9zxjcyY5q4heNaSjunlCPGazu7Zqi

    > oBl02KxwU1SYvvb9mY+WOhj12jksduKiohslOI+/sadfglsiald

    > -----END RSA PRIVATE KEY-----

    > EOF

    The value returned must be identical to the value calculated in Step 1.

    If the values do not match, then your PEM key may not be entered correctly, or you have not entered the correct PEM key on your server. Check your license information again to ensure you have not missed a step.

     

    Check NTP Connectivity


    NTP is rarely the cause of license validation problems, and in most cases it is transient. If your organization limits NTP connectivity, you can provide the information returned to your network administrators for assistance. In the returned output, the NTP service is connected and your VIRL server has established a peer. If they all show INIT consult with your network admin for assistance with NTP server connectivity. In some environments external NTP server connectivity is not permitted.

    ntpq -pn

     

    The returned output should be similar to this: (note: * indicates NTP peer)

    virl@sim1:~$ ntpq -pn

        remote          refid      st t when poll reach  delay  offset  jitter

    ==============================================================================

    +204.2.134.164  199.233.236.226  3 u  12  128  377  82.570    0.337  3.784

    -63.211.239.58  132.163.4.103    2 u    5  128  377  85.638  -13.518  1.629

    *204.11.201.12  131.107.13.100  2 u  74  128  377  95.083    0.263  1.037

    -142.54.181.202  64.113.32.5      2 u    6  128  377  50.530  -4.383  2.423

    +52.6.160.3      209.51.161.238  2 u    1  128  377  20.068  -0.954  1.382


    If the command returned no peers, test connectivity using ntpdate. The returned output should show an exchange between the VIRL server and destination NTP server.

    virl@sim1:~$ ntpdate -d 204.11.201.12

     

    For more information about troubleshooting NTP errors, take a look at T-Shoot: NTP errors and connectivity on VIRL

     

    Requesting Assistance

    If authentication continues to fail, clear all fields from Salt Configuration and click Reset. After the page refreshes, re-enter your license information as outlined above based on your VIRL version. If the problem continues, please collect the output from the commands listed here or run VIRL Server Config Validation script and attach the output text file.