Port from SW - Salt master rsa key issue
-
@DustinB3403 said in Port from SW - Salt master rsa key issue:
@scottalanmiller said in Port from SW - Salt master rsa key issue:
@DustinB3403 said in Port from SW - Salt master rsa key issue:
@dgingerich Hrm. . .
If you're just entering through the process I don't think it would be the RSA keys then . . . maybe there is a firewall enabled on your Masters/Minions?
Given that it worked and the key regen broke it, it's safe to assume it's a key issue.
I was under the assumption he replaced all of the keys.
Right, that is the break.
-
One of our big Salt users is @QuixoticJeremy and he is at the MangoMeetup event today.
-
I'm trying to research this, but this is definitely not a common issue.
-
@scottalanmiller said in Port from SW - Salt master rsa key issue:
I'm trying to research this, but this is definitely not a common issue.
Perhaps he should contact vendor support?
-
I am spinning up an additional system to try the "install salt, connect them, confirm communication, generate rsa keys, confirm disconnect" method. After that, I'll try generating the rsa keys before installing salt and see if that makes any difference. (I hate spinning up most systems, as they cost my company money to just start them up. I start up one, test on it, and delete it a day later, it still costs my company $36.50. So, this test will cost us $73.)
-
What is the contents of your PKI folder, like this...
# ll /etc/salt/pki/master/ total 28 -r-------- 1 root root 1674 Dec 16 2016 master.pem -rw-r--r-- 1 root root 450 Dec 16 2016 master.pub drwxr-xr-x 2 root root 4096 Jun 14 21:00 minions drwxr-xr-x 2 root root 4096 Dec 16 2016 minions_autosign drwxr-xr-x 2 root root 4096 Mar 19 16:26 minions_denied drwxr-xr-x 2 root root 4096 Jun 14 21:00 minions_pre drwxr-xr-x 2 root root 4096 Dec 16 2016 minions_rejected -
@scottalanmiller said in Port from SW - Salt master rsa key issue:
What is the contents of your PKI folder, like this...
# ll /etc/salt/pki/master/ total 28 -r-------- 1 root root 1674 Dec 16 2016 master.pem -rw-r--r-- 1 root root 450 Dec 16 2016 master.pub drwxr-xr-x 2 root root 4096 Jun 14 21:00 minions drwxr-xr-x 2 root root 4096 Dec 16 2016 minions_autosign drwxr-xr-x 2 root root 4096 Mar 19 16:26 minions_denied drwxr-xr-x 2 root root 4096 Jun 14 21:00 minions_pre drwxr-xr-x 2 root root 4096 Dec 16 2016 minions_rejectedYes, the contents of my pki folder look just like that, except with different dates.
root@QAICS-MAN-01:/etc/salt/pki/master# ls -l
total 28
-r-------- 1 root root 1674 Jun 23 18:17 master.pem
-rw-r--r-- 1 root root 450 Jun 23 18:17 master.pub
drwxr-xr-x 2 root root 4096 Jun 23 18:35 minions
drwxr-xr-x 2 root root 4096 Jun 23 18:17 minions_autosign
drwxr-xr-x 2 root root 4096 Jun 23 18:17 minions_denied
drwxr-xr-x 2 root root 4096 Jun 23 18:35 minions_pre
drwxr-xr-x 2 root root 4096 Jun 23 18:17 minions_rejected
root@QAICS-MAN-01:/etc/salt/pki/master# -
What are the date times for the first two?
-
@scottalanmiller said in Port from SW - Salt master rsa key issue:
What are the date times for the first two?
update previous post with that info
-
Of course, trying the same sequence, I cannot reproduce the results. Looks like I'm going to have to rebuild the masters.
Maybe copying the rsa key files to the new systems will be possible.
-
@dgingerich said in Port from SW - Salt master rsa key issue:
Of course, trying the same sequence, I cannot reproduce the results. Looks like I'm going to have to rebuild the masters.
Maybe copying the rsa key files to the new systems will be possible.
Possible. Or it might be worth accepting the pain of changing the keys on GIT.
-
rebuilding the systems did not work. getting the same issue with brand new master under the same name.
-
@dgingerich said in Port from SW - Salt master rsa key issue:
rebuilding the systems did not work. getting the same issue with brand new master under the same name.
You didn't do anything with the keys, you left the new system with its automatically created keys? The minions will not be able to rejoin with the same name, you'll need to remove them and add them again.
-
@scottalanmiller The other minions were not included in this. The masters would not even log into themselves after the keys were accepted. Total blank slate machines, install salt-master and salt-minion, configure them to point to themselves, (other minions were turned off) and they still failed the test.ping and salt-minion -l debug gave the exact same result. Could it be something with the DNS info?
-
So the masters can't see THEMSELVES? What master name are you using? You could try using a straight IP address to test. Yes, DNS could do this.
-
root@QAICS-MAN-01:~# apt-get install salt-master salt-minion
...
root@QAICS-MAN-01:/etc/salt# vi minion
root@QAICS-MAN-01:/etc/salt# vi master
root@QAICS-MAN-01:/etc/salt# service salt-master start
root@QAICS-MAN-01:/etc/salt# service salt-minion start
root@QAICS-MAN-01:/etc/salt# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
QAICS-MAN-01
QAICS-MAN-02
Rejected Keys:
root@QAICS-MAN-01:/etc/salt# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
QAICS-MAN-01
QAICS-MAN-02
Rejected Keys:
root@QAICS-MAN-01:/etc/salt# salt-key -A
The following keys are going to be accepted:
Unaccepted Keys:
QAICS-MAN-01
QAICS-MAN-02
Proceed? [n/Y] y
Key for minion QAICS-MAN-01 accepted.
Key for minion QAICS-MAN-02 accepted.
Key for minion QAICS-Proxy-01 accepted.
root@QAICS-MAN-01:/etc/salt# salt-key -L
Accepted Keys:
QAICS-MAN-01
QAICS-MAN-02
Denied Keys:
Unaccepted Keys:
Rejected Keys:
root@QAICS-MAN-01:/etc/salt# salt '*' test.ping
[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
[WARNING ] jid does not exist
[WARNING ] Returner unavailable:
QAICS-MAN-02:
Minion did not return. [No response]
QAICS-MAN-01:
Minion did not return. [No response]
root@QAICS-MAN-01:/etc/salt# -
@dgingerich said in Port from SW - Salt master rsa key issue:
[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
maybe you have duplicate DNS entries and round robin is getting you?
-
@scottalanmiller said in Port from SW - Salt master rsa key issue:
@dgingerich said in Port from SW - Salt master rsa key issue:
[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
maybe you have duplicate DNS entries and round robin is getting you?
The DNS is just one address for each. I changed the to the new IPs while the systems were building at the authoritative servers. (Systems are housed at Packet.net, DNS handled through AWS) So, there should not be any DNS caching issues.
-
@dgingerich said in Port from SW - Salt master rsa key issue:
@scottalanmiller said in Port from SW - Salt master rsa key issue:
@dgingerich said in Port from SW - Salt master rsa key issue:
[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
maybe you have duplicate DNS entries and round robin is getting you?
The DNS is just one address for each. I changed the to the new IPs while the systems were building at the authoritative servers. (Systems are housed at Packet.net, DNS handled through AWS) So, there should not be any DNS caching issues.
Oh okay, this is all hosted. Still, best to be sure and rule out possibilities while testing. This is weird, we use Vultr for Salt Masters and have never seen anything like this. But we avoid Ubuntu, so if there is any bug there, we'd not have seen it.
-
@scottalanmiller said in Port from SW - Salt master rsa key issue:
@dgingerich said in Port from SW - Salt master rsa key issue:
@scottalanmiller said in Port from SW - Salt master rsa key issue:
@dgingerich said in Port from SW - Salt master rsa key issue:
[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
maybe you have duplicate DNS entries and round robin is getting you?
The DNS is just one address for each. I changed the to the new IPs while the systems were building at the authoritative servers. (Systems are housed at Packet.net, DNS handled through AWS) So, there should not be any DNS caching issues.
Oh okay, this is all hosted. Still, best to be sure and rule out possibilities while testing. This is weird, we use Vultr for Salt Masters and have never seen anything like this. But we avoid Ubuntu, so if there is any bug there, we'd not have seen it.
I was able to build another system, named QAICS-mastertest, that worked perfectly using the exact same methods. It's really weird.
