4 Replies Latest reply: Aug 15, 2019 2:55 PM by Karlo Bobiles RSS

    Salt status failing - failed connection to servers

    Martin

      Hello,

      I'm aware of the issues with connecting to the 4 (EU in my case) authentication servers and the proposed solutions:

      - Leaving only 1 server specified in the salt config

      - changing the mtu to 1000 on eth0 connecting to the physical network

       

      However, none of these workarounds seem to do any magic, the results are still the same - Connect = Failed (with the former checks passing).

      We're running v1.5 and will most likely be upgrading to 1.6 in the upcoming days/weeks, but this issue takes priority since we can't fire up new labs in VIRL.

       

      Is there anything we can do, or is there any (other) workaround for this?

       

      Thanks in advance,

      Martin.

        • 1. Re: Salt status failing - failed connection to servers
          Karlo Bobiles

          Hello Martin,

           

          Can you please run the VIRL Server Salt Connectivity Validation script and upload the results to this thread? This will give me a better idea.

           

          I want to verify the status of your license.

           

          Thank you,
          Karlo Bobiles

          Cisco Learning Network

          • 2. Re: Salt status failing - failed connection to servers
            Martin

            Hello Karlo,

             

            Sure thing. Output below:

             

            virl@virl:~$ ./slt_tstr.sh -c -v

             

             

            Testing Connectivity to: [vsm-eu-51.virl.info 173.38.221.79]

            Connection to vsm-eu-51.virl.info 4505 port [tcp/*] succeeded!

            Connection to vsm-eu-51.virl.info 4506 port [tcp/*] succeeded!

             

             

            Checking License....[ 8D5461C3.virl.info ]

            Auth test --> Salt Server [ vsm-eu-51.virl.info ]

            [DEBUG   ] Reading configuration from /etc/salt/minion

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/extra.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/extra.conf

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/openstack.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/openstack.conf

            [DEBUG   ] Configuration file path: /etc/salt/minion

            [WARNING ] Insecure logging configuration detected! Sensitive data may be logged.

            [DEBUG   ] Reading configuration from /etc/salt/minion

            [WARNING ] Insecure logging configuration detected! Sensitive data may be logged.

            [DEBUG   ] Reading configuration from /etc/salt/minion

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/extra.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/extra.conf

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/openstack.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/openstack.conf

            [DEBUG   ] Connecting to master. Attempt 1 of 1

            [DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', '8D5461C3.virl.info', 'tcp://173.38.221.79:4506')

            [DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (8215)

            [DEBUG   ] Setting zmq_reconnect_ivl to '8215ms'

            [DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'

            [DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '8D5461C3.virl.info', 'tcp://173.38.221.79:4506', 'clear')

            [CRITICAL] The Salt Master has rejected this minion's public key!

            To repair this issue, delete the public key for this minion on the Salt Master and restart this minion.

            Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit.

             

             

            Testing Connectivity to: [vsm-eu-52.virl.info 173.38.221.79]

            Connection to vsm-eu-52.virl.info 4505 port [tcp/*] succeeded!

            Connection to vsm-eu-52.virl.info 4506 port [tcp/*] succeeded!

             

             

            Checking License....[ 8D5461C3.virl.info ]

            Auth test --> Salt Server [ vsm-eu-52.virl.info ]

            [DEBUG   ] Reading configuration from /etc/salt/minion

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/extra.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/extra.conf

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/openstack.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/openstack.conf

            [DEBUG   ] Configuration file path: /etc/salt/minion

            [WARNING ] Insecure logging configuration detected! Sensitive data may be logged.

            [DEBUG   ] Reading configuration from /etc/salt/minion

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/extra.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/extra.conf

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/openstack.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/openstack.conf

            [DEBUG   ] Connecting to master. Attempt 1 of 1

            [DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', '8D5461C3.virl.info', 'tcp://173.38.221.79:4506')[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', '8D5461C3.virl.info', 'tcp://173.38.221.79:4506')

            [DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (4595)

            [DEBUG   ] Setting zmq_reconnect_ivl to '4595ms'

            [DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'

            [DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '8D5461C3.virl.info', 'tcp://173.38.221.79:4506', 'clear')

            [CRITICAL] The Salt Master has rejected this minion's public key!

            To repair this issue, delete the public key for this minion on the Salt Master and restart this minion.

            Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit.

             

             

            Testing Connectivity to: [vsm-eu-53.virl.info 173.38.221.80]

            Connection to vsm-eu-53.virl.info 4505 port [tcp/*] succeeded!

            Connection to vsm-eu-53.virl.info 4505 port [tcp/*] succeeded!

            Connection to vsm-eu-53.virl.info 4506 port [tcp/*] succeeded!

             

             

            Checking License....[ 8D5461C3.virl.info ]

            Auth test --> Salt Server [ vsm-eu-53.virl.info ]

            [DEBUG   ] Reading configuration from /etc/salt/minion

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/extra.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/extra.conf

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/openstack.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/openstack.conf

            [DEBUG   ] Configuration file path: /etc/salt/minion

            [WARNING ] Insecure logging configuration detected! Sensitive data may be logged.

            [DEBUG   ] Reading configuration from /etc/salt/minion

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/extra.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/extra.conf

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/openstack.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/openstack.conf

            [DEBUG   ] Connecting to master. Attempt 1 of 1

            [DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', '8D5461C3.virl.info', 'tcp://173.38.221.80:4506')

            [DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (4210)

            [DEBUG   ] Setting zmq_reconnect_ivl to '4210ms'

            [DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'

            [DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '8D5461C3.virl.info', 'tcp://173.38.221.80:4506', 'clear')

            [CRITICAL] The Salt Master has rejected this minion's public key!

            To repair this issue, delete the public key for this minion on the Salt Master and restart this minion.

            Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit.

             

             

            Testing Connectivity to: [vsm-eu-54.virl.info 173.38.221.80]

            Connection to vsm-eu-54.virl.info 4505 port [tcp/*] succeeded!

            Connection to vsm-eu-54.virl.info 4506 port [tcp/*] succeeded!

             

             

            Checking License....[ 8D5461C3.virl.info ]

            Auth test --> Salt Server [ vsm-eu-54.virl.info ]

            [DEBUG   ] Reading configuration from /etc/salt/minion

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/extra.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/extra.conf

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/openstack.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/openstack.conf

            [DEBUG   ] Configuration file path: /etc/salt/minion

            [WARNING ] Insecure logging configuration detected! Sensitive data may be logged.

            [DEBUG   ] Reading configuration from /etc/salt/minion

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/extra.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/extra.conf

            [DEBUG   ] Including configuration from '/etc/salt/minion.d/openstack.conf'

            [DEBUG   ] Reading configuration from /etc/salt/minion.d/openstack.conf

            [DEBUG   ] Connecting to master. Attempt 1 of 1

            [DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', '8D5461C3.virl.info', 'tcp://173.38.221.80:4506')

            [DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (3886)

            [DEBUG   ] Setting zmq_reconnect_ivl to '3886ms'

            [DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'

            [DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', '8D5461C3.virl.info', 'tcp://173.38.221.80:4506', 'clear')

            [CRITICAL] The Salt Master has rejected this minion's public key!

            To repair this issue, delete the public key for this minion on the Salt Master and restart this minion.

            Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit.

            ----------------------------------------------------------

             

            Looking at the output, I decided to look at the invoices to make sure the licence is actually active (despite being bought several months ago as far as I know), but there is no trace of our licence purchase, the last invoice I'm seeing is from 11/23/2017 (which clearly can't be true, since our team was using virl for the last few months flawlessly).

            I'm currently waiting for an accountable person who bought the licence to confirm this.

             

            I should have an answer in the upcoming hour or two.

             

            Thank you for you time, Karlo.

            • 3. Re: Salt status failing - failed connection to servers
              Martin

              Okay, the accountable person is out of office till 19th Aug.

               

              Despite that, I'm positive there was an active licence that the person bought, we were working on it and were resetting it twice I think when we experienced problems with connection to the servers - everything worked.

              • 4. Re: Salt status failing - failed connection to servers
                Karlo Bobiles

                Hello Martin,

                 

                Our records indicate your license has expired as of  2018-11-23. In order to continue using VIRL, you'll need to purchase a new license. No need to reinstall, once you've purchased a new license, you can apply that to your server.

                 

                Thank you,

                Karlo Bobiles
                Cisco Learning Network