I've inherited a XenServer 6.2 environment. My background is in Hyper-V. I understand some concepts, but I have not used these, or the Dell PowerVault SAN that it's attached to via iSCSI, so bear with me if I am lacking in background.

Current Status - XenPool with two hosts, Xen1 and Xen2 (Pool Master). VMs show as running, but can't interact with them in any way. All forms of storage are disconnected from the hosts. When I ran iscsiadmn node and session, from the console, Xen2 showed as being connected to the SAN. However, I cannot connect either host to their local storage. Both stuck in maintenance mode.

Background, and of course, I can flesh it out more, as questions arise. I'm going on 26 hours of work, -4 hours of sleep. Forgive any holes. I walked in yesterday to do maintenance when no one was here. I noticed I couldn't RDP into the servers that were managed by Xen1. I logged in, and not being familiar enough, I didn't notice if the storage connectivity was an issue right away. At the time, it seemed like there were networking issues. I couldn't force down any VMs, I couldn't move any VMs to Xen2. Tried to reboot the host, couldn't reboot it from XenCenter. Eventually had to force the server down. Over time, I saw that there was a with the control domain getting full, at 91%. Doing research, I cleared out some log files to get it under 3GB.

I repeatedly had issues with Xen1 going into maintenance mode after a reboot, and not being able to exit. Only resolved when I changed anything about its network settings. Eventually, I was commanded to restart both servers. After they came back up, Xen2 also lost all connection to its storage, and is now stuck in maintenance mode. Which means we have no servers at all, including Active Directory. I have internet by pointing DNS to my router.

Help!

@scottalanmiller Thank you. I've been an IT professional, or in IT leadership (but still hands-on) since the 90's. I took on a role a few months ago with a organization that's been patched all to heck and back, and I've been fighting crisis after crisis to try to make them fundamentally sound. I'm primarily a Windows/AD/Networking guy, and Scott brought me here as a resource for my latest crisis, XenServer.

I'm Dallas based, a husband and father, and I have a passion for worldwide travel. If you're looking to find a deal to get somewhere, I'm a good resource to have in your pocket.

On the phone with Dell, and the logs indicate that the two drives went down within seconds of each other, which may indicate that one of the drives isn't truly dead. They're going to try to revive it. Fingers crossed.

@scottalanmiller He's been talking to his most trusted employee, a developer. Apparently he doesn't even understand that that Xen and the SAN are two completely different things...

I do plan on updating my LinkedIn profile and resume over the next few days.

@NerdyDad Close. Friend of the family. He's also a really good guy, and probably a genius. He's on my side. Not necessarily a correlation to his genius, just sayin'.

@scottalanmiller

@scottalanmiller said in If you are new drop in say hello and introduce yourself please!:

@dominica and I are full time world travelers. Based in Dallas more than anywhere else in the US (own a home there) we've been living outside of the US for the last two years. Have been to 31 countries while living out of Spain, Panama, Nicaragua, Greece (Crete) and Romania. We leave for a week's vacation in Rome in 19 days and from there going to our new home in Noto, Siciliy.

I'll be in Venice for a couple of days mid-January, and taking the family to Rome for a week in mid-February.

This was my 2016 travel. http://www.gcmap.com/mapui?P=dfw-clt-jfk-auh%2C+dxb-doh-bkk-hkg-hnd-sfo-dfw dfw-msy-dfw dfw-sfo-dfw dfw-lax-dfw dfw-las-dfw dfw-las-dfw dfw-san-dfw dfw-lax-hnd-sfo-lax-dfw dfw-lax-syd-sin-icn-sfo-dfw dfw-lax-syd-dfw dfw-lax-jfk-mia-pty-mia-jfk-lax-aus-dfw dfw-las-dfw dfw-yvr-phx-san-mia-pty-mia-dfw dfw-dca-dfw dfw-aus-dfw dfw-jfk-zrh-lhr-dfw dfw-lhr-cdg-dfw tyr-dfw-nrt-hkg-dfw-tyr&MS=wls&DU=mi

@scottalanmiller I had gone to another user's machine to log into the PowerVault interface last night. I'm installing the software on my machine now.

Also, we're willing to engage someone formally to get this resolved.

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

I think that we need to look at the logs. Do you have access to get onto the machines themselves via SSH?

I haven't tried yet. Let me get a hold of putty.

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

I think that we need to look at the logs. Do you have access to get onto the machines themselves via SSH?

I'm logged into Xen2 with SSH now.

@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Once this crisis is over, I recommend going over this when you have free time.

https://mangolassi.it/topic/7825/sam-learning-linux-system-administration

Its not complete but its a really good start.

Learning Linux has always been something I've wanted to do, but have never been able to find the time to do so in a work environment.

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Everything that ends in .gz can be safely just moved off (deleting from the source) as they are archival and not being used.

rm /var/log/*.gz
Is that correct?

@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

What is the network setup on these servers and the SAN?

One switch, multiple switches? how many connections to each server? etc.

@NerdyDad On the plus side, this environment was so effed up, I'm not THAT terribly upset about starting from scratch. It'll be a beating to get a lot of that data back, however. Anyone know of any services able to accomplish such a thing?

@scottalanmiller

At this point, I'm even at a loss on trying to figure out where to begin to explain to him how this is possible, as he hasn't seem to accept the premise in the first place.

First, thanks everyone for your help yesterday, here's a brief update, more to come after a couple of phone calls.
I've powered down the SAN and reseated the drives. No joy. About to call Dell and see what options I may have, if any. The good news is there's a second SAN, and some data for the VMs is on there, but few OSes. The good news, is that somehow, I really have no idea how, the Xen servers reattached to the local storage overnight, and to the one virtual disk (Spindle2) and one DC is available. I'll give more details later.

I wouldn't blame you for thinking I'm an incompetent tool, but I appreciate your help, anyway.

@Dashrender

No, there was a second SAN of which I was unaware. I know how bad that sounds, but since I've started here, it's been crisis after crisis, and I've had a quote from Scale that would have made all of this obsolete, provided the owner approved it in a conversation next week. I'm finding new things every day. With this probably going away in a week, I didn't spend a lot of time on it. This entire environment has been triage since I walked in the door.

And yes, a cobbled together DC is in the plans after I get off the horn with Dell.

@Dashrender @scottalanmiller Systems are up. New disk in, VD is degraded and being rebuilt.