HowTo: Upgrade Salt-Minions and Don't Lose Them

If you try to upgrade salt-minions from the salt-master, in most cases salt-master would probably lose access to the minions.

If the access is lost, you would have to log in to each host and restart the salt-minion service manually to make it communicate back with the salt-master.

This problem is caused by the salt-minion service that often doesn't restart after update.

In this article i'll show how to safely update salt-minions from the salt-master without loosing communication.

Upgrade Salt-Master

First of all it is required to upgrade salt-master.

According to documentation, the master should always be upgraded first, as the backward compatibility for minions running newer versions of salt than their masters is not guaranteed.

Upgrade salt-master on CentOS, RHEL, etc.:
$ sudo yum update salt-master
Upgrade salt-master on Debian, Ubuntu, Linux Mint, etc.:
$ sudo apt-get update salt-master

To find out the current version of the salt-master, run:

$ salt-master --version

Upgrade Salt-Minion

For the moment there is no a normal way to upgrade salt-minions from the salt-master and not to lose communication between them.

But you can create a salt-state, that will send a special shell command to the salt-minion and execute it in the background.

This shell command will update and restart the salt-minion service using the locally executed salt-call.

After the salt-minion is restarted, communication with the salt-master would be successfully recovered.

At the same time the below state beforehand checks if a new version of the salt-minion packaged is available to avoid useless restart of the salt-minion service.

Create the following state on the salt-master and apply it to the salt-minions to upgrade them safely:

Upgrade Salt-Minion:
    - name: |
        exec 0>&- # close stdin
        exec 1>&- # close stdout
        exec 2>&- # close stderr
        nohup /bin/sh -c 'salt-call --local pkg.install salt-minion && salt-call --local service.restart salt-minion' &
    - onlyif: "[[ $(salt-call --local pkg.upgrade_available salt-minion 2>&1) == *'True'* ]]" 

To find out the current version of your salt-minions from the salt-master, run:

$ sudo salt '*' test.version