T-Shoot: NTP errors and connectivity on VIRL

     

     

    Summary

    In some environments NTP (Network Time Protocol) may be restricted to an internally managed server or limited to pre-determined public NTP servers. These definitions as set by the system administrator to meet company security guidelines or general policy. In most cases, all public NTP servers are allowed but there is no way to guarantee NTP service will be available to VIRL virtual machine when deployed. This guide will assist in finding the cause of common NTP sync errors and provide steps to mediate.

     

    Getting Started

    To troubleshoot NTP connectivity you will need to run commands directly on your VIRL server. Start two separate SSH connections and log in.

    Default Credentials:

    User name: virl

    Password: VIRL

     

    Set your timezone

    • Using one of the SSH connections set your timezone
      • sudo dpkg-reconfigure tzdata
      • Use the curser and tab keys to navigate and spacebar to select. [Enter] confirms your selection.
    • Wait at least 2 minutes and then check for peers
      • sudo ntpq -p
      • If after 5 minutes output still reports 'INIT' continue

     

    Testing Connectivity to NTP Servers

    For this test, we must stop the NTP service. If the service is not stopped, some commands will fail because the service will not release UDP port 123.

    • Stop NTP service
      • sudo service ntp stop
    • From second ssh session start a packet capture
      • sudo tcpdump -ni eth0 udp port 123
    • From first ssh session, test access to default pool
      • sudo ntpdate -d pool.ntp.org
    • Below is the expected result from the above command (viewed from first ssh session)

     

    virl@virldev-5058:~$ sudo ntpdate -d pool.ntp.org
    24 Mar 13:10:15 ntpdate[7875]: ntpdate 4.2.6p5@1.2349-o Wed Oct  5 12:35:26 UTC 2016 (1)
    Looking for host pool.ntp.org and service ntp
    host found : ha81.smatwebdesign.com
    transmit(198.58.105.63)
    receive(198.58.105.63)
    transmit(108.61.73.243)
    receive(108.61.73.243)
    …
    server 198.58.105.63, port 123
    stratum 2, precision -23, leap 00, trust 000
    refid [198.58.105.63], delay 0.06229, dispersion 0.00005
    transmitted 4, in filter 4
    reference time:    dc7fd178.97ffc68f  Fri, Mar 24 2017 13:10:16.593
    originate timestamp: dc7fd17e.218f19ce  Fri, Mar 24 2017 13:10:22.131
    transmit timestamp:  dc7fd17e.1c96472e  Fri, Mar 24 2017 13:10:22.111
    filter delay:  0.06270  0.06229  0.06233  0.06236
            0.00000  0.00000  0.00000  0.00000
    filter offset: 0.001138 0.001076 0.000975 0.001024
            0.000000 0.000000 0.000000 0.000000
    delay 0.06229, dispersion 0.00005
    offset 0.001076
    
    server 108.61.73.243, port 123
    stratum 2, precision -20, leap 00, trust 000
    refid [108.61.73.243], delay 0.03885, dispersion 0.00002
    transmitted 4, in filter 4
    reference time:    dc7fcc15.313c88e2  Fri, Mar 24 2017 12:47:17.192
    originate timestamp: dc7fd17e.522f59a0  Fri, Mar 24 2017 13:10:22.321
    transmit timestamp:  dc7fd17e.4fcaf481  Fri, Mar 24 2017 13:10:22.311
    filter delay:  0.03917  0.03896  0.03893  0.03885
            0.00000  0.00000  0.00000  0.00000
    filter offset: 0.002846 0.002639 0.002692 0.002678
            0.000000 0.000000 0.000000 0.000000
    delay 0.03885, dispersion 0.00002
    offset 0.002678
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    

     

    • Note that in the start of the output we see both 'transmit' and 'receive'; this is the expected result and clearly shows connectivity.
    • Next output shows that no response from NTP servers. Until this behavior is corrected you will not be able to sync your date/time information.

     

    virl@virldev-5058:~$ sudo ntpdate -d pool.ntp.org
    2 Dec 14:08:16 ntpdate[19395]: ntpdate 4.2.6p5@1.2349-o Wed Oct  5 12:35:26 UTC 2016 (1)
    Looking for host pool.ntp.org and service ntp
    host found : ha81.smatwebdesign.com
    transmit(198.58.105.63)
    transmit(108.61.73.243)
    198.58.105.63: Server dropped: no data
    server 198.58.105.63, port 123
    stratum 0, precision 0, leap 00, trust 000
    refid [198.58.105.63], delay 0.00000, dispersion 64.00000
    transmitted 4, in filter 4
    reference time:    00000000.00000000  Sun, Dec 31 1899 19:00:00.000
    originate timestamp: 00000000.00000000  Sun, Dec 31 1899 19:00:00.000
    transmit timestamp:  dbec4526.57319f9f  Fri, Dec  2 2016 14:08:22.340
    filter delay:  0.00000  0.00000  0.00000  0.00000
            0.00000  0.00000  0.00000  0.00000
    filter offset: 0.000000 0.000000 0.000000 0.000000
            0.000000 0.000000 0.000000 0.000000
    delay 0.00000, dispersion 64.00000
    offset 0.000000
    
    

     

    • If your returned output is similar to above section, we recommend configuring another NTP provider (server) which may resolve the problem. Continue to section Resetting NTP Servers via UWM OR...
    • Collect the output generated in the second ssh session and provide both outputs to your system administrator for assistance. If your network does not have a system administrator and you would like assistance via the forum, collect the output and start a new thread. Be sure to include details of your deployment and steps you have already taken.

     

    Resetting NTP Servers via UWM

    • Log into UWM web interface with admin user
    • Default Credentials
      • Username: uwmadmin
      • Password: password
    • Navigate to VIRL Server > System Configuration
    • Change NTP Server from currently defined (default is: pool.ntp.org) to appropriate server as required by your network. If the default server is not working and there are no restrictions on NTP at your location, try any of the following public NTP servers:
    • Do not edit any files via CLI. Just replace the NTP server defined in UWM with the name of your chosen provider. For example setting the NTP Server to time.nist.gov,  allows VIRL to use all of the NIST time servers defined in their pool.
    • Apply Setting and follow prompts
    • Test connectivity as outlined in Testing Connectivity to NTP Servers above

     

    Resetting NTP Servers via CLI  (Not Recommended)

    • Edit virl.ini to reflect desired NTP server pool
    • If your environment uses a proxy server, edit virl.ini to reflect correct proxy server
    • Edit /etc/ntp.conf to reflect desired NTP server pool (must match NTP server pool defined in virl.ini)
    • Run following commands to update VIRL settings
      • sudo salt-call -l debug state.sls virl.vinstall
      • sudo vinstall salt
      • sudo salt-call -l debug state.sls virl.ntp