Salt Minion Problems

scottalanmiller

I'm rebooting to test something. For some reason, the Salt Master never showed up in the key list either. This implies something is certainly wrong.

Romo

@scottalanmiller said in Salt Minion Problems:

I'm rebooting to test something. For some reason, the Salt Master never showed up in the key list either. This implies something is certainly wrong.

Exactly thats what I ment

@scottalanmiller said in Salt Minion Problems:

@Romo said in Salt Minion Problems:

@scottalanmiller shouldn't the salt-master have already sent its own key? In my vm the first key I accepted was the key my master.

This could mean there is something else that's wrong don't you think?

Which part? The master? That seems fine.

Romo

@WrCombs By the way, do you remember if you did open the ports in the master?

Romo

All set on my install, I hadn't reloaded the firewall rules so that's why my minion could reach the master.

0_1485894347865_Screenshot from 2017-01-31 14:25:31.png

scottalanmiller

@Romo said in Salt Minion Problems:

@WrCombs By the way, do you remember if you did open the ports in the master?

He did, I looked.

scottalanmiller

Here is his firewall...

# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources: 
  services: dhcpv6-client ssh
  ports: 4505-4506/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  sourceports: 
  icmp-blocks: 
  rich rules:

scottalanmiller

Here is the error message on the Salt Master's minion process:

Jan 31 20:32:37 salt salt-minion[901]: [ERROR   ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

Romo

@scottalanmiller said in Salt Minion Problems:

The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

That happens to all minions when they are waiting for the master to approve their key.

scottalanmiller

@Romo said in Salt Minion Problems:

@scottalanmiller said in Salt Minion Problems:

The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

That happens to all minions when they are waiting for the master to approve their key.

Yeah, the master didn't have an /etc/hosts entry so it was pointing itself to a different Salt Master. So that was awfully confusing.

scottalanmiller

Ok, so after fixing the hostname on the master, fixing the /etc/hosts file and then checking the errors, we find that that weird ip local address there is NOT from the Minion, but from the Master itself. So the Minion has never checked in at all.

# salt-key --list-all
Accepted Keys:
Denied Keys:
ip-65-75-137-152.local
Unaccepted Keys:
ip-65-75-137-152.local
Rejected Keys:
[root@lab-lnx-william-salt ~]# systemctl status salt-minion
● salt-minion.service - The Salt Minion
   Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Tue 2017-01-31 20:39:37 UTC; 2min 43s ago
  Process: 902 ExecStart=/usr/bin/salt-minion (code=exited, status=0/SUCCESS)
 Main PID: 902 (code=exited, status=0/SUCCESS)

Jan 31 20:39:22 lab-lnx-william-salt systemd[1]: Starting The Salt Minion...
Jan 31 20:39:23 lab-lnx-william-salt systemd[1]: Started The Salt Minion.
Jan 31 20:39:37 lab-lnx-william-salt salt-minion[902]: [CRITICAL] The Salt Master has rejected this minion's public key!
Jan 31 20:39:37 lab-lnx-william-salt salt-minion[902]: To repair this issue, delete the public key for this minion on the Salt Master and restart this minion.
Jan 31 20:39:37 lab-lnx-william-salt salt-minion[902]: Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit.

scottalanmiller

Here is the current error on the minion. I'm guessing we have the same issue:

[root@minion1 ~]# systemctl status salt-minion
● salt-minion.service - The Salt Minion
   Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-01-31 20:21:59 UTC; 23min ago
 Main PID: 1228 (salt-minion)
   CGroup: /system.slice/salt-minion.service
           ├─1228 /usr/bin/python /usr/bin/salt-minion
           ├─1237 /usr/bin/python /usr/bin/salt-minion
           └─1241 /usr/bin/python /usr/bin/salt-minion

Jan 31 20:44:27 minion1 salt-minion[1228]: [ERROR   ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

scottalanmiller

Here are some issues in the /etc/hosts file that will definitely cause things to fail:

192.168.1.97



192.168.1.94 salt
192.128.1.94 salt

Dashrender

All this because you use another DNS that's not part of the project and it has overlapping details for other testers/production. lol

scottalanmiller

@Dashrender said in Salt Minion Problems:

All this because you use another DNS that's not part of the project and it has overlapping details for other testers/production. lol

No, you would need this to be correct either way.

Romo

@scottalanmiller check what does salt have as the minion_id as well, it is in /etc/salt/minion_id

scottalanmiller

Also found the /etc/hosts file had a full double copy of itself in the same file. No idea how that got there.

scottalanmiller

Salt Master will not allow me to delete the keys that are in there. I did a delete all and it said that it was, but it failed.

WrCombs

okay. what do i need to do to fix it? sorry for the headache guys

WrCombs

well it looks like Scott fixed most of the problems. ( ish ) I just logged on to the server and all the things he pointed out as wrong, arent there anymore

scottalanmiller

@WrCombs said in Salt Minion Problems:

okay. what do i need to do to fix it? sorry for the headache guys

No idea.

Unsolved Salt Minion Problems