No needs to buy anything: you just spin a Linux VM with enough storage your server, and build a time machine server compatible service using Netatalk and Avahi. I've used this from years till today, it's perfectly realiable and very simple; it's enterprise-ready also, you can made this VM as big as you want, backup to tape or replicate as any other VM, and of course host it on enterprise-class hardware etc.

I've read somewhere that Apple's TM is just a linux or bsd machine with a tuned version of netatalk/avahi.
I've build my own "Time Capsule" in a CentOS VM hosted in a KVM environment, no issue at all from 2013. Just search "linux time machine netatalk" and you will find a bunch of tutorials.

@sreekumarpg
Mount S3 as a filesystem with https://github.com/archiecobbs/s3backer , and use your standard backup tools.

Scott, let me tell that you're completely right.
I use vi as my primary editor, with little customization.
Vi is everywhere, from the ESXi busybox to router firmware CLI… and I really like how it works.
My only non-standard temptation is ZSH, but I usually can do almost everything with plain bash (set -o vi, of course).

@scottalanmiller said:

Putting a non-OEM disk into a hot swap bay is no different then putting on into a PCIe bay.

Errr, no, It will lack the caddy, and I don't think they sell it as spare parts.

@scottalanmiller

@scottalanmiller said:

@Francesco-Provino said:

Unfortunately, this is not my case: OEM SSD aren't supported with our RAID cards in the servers, and VMware can't do software RAID (apart from, well, sort of, uhm, VSAN).
IBM's SAS SSD are still incredibly expensive.

Ah... the devil is in the details. You are using VMware and lack enterprise software RAID options so can't do super high performance SSD without having a RAID card to support it. Yet another VMware caveat. They screw you at every turn. So many limitations that you would never guess would be there.

Are you sure that "unsupported" is the case, though? Of course it is not supported by IBM, neither is the Intel PCIe board. So both cases are equally without support. The question is "do they work?"

This consideration is interesting, but I don't even know if it's possible to put OEM disks in those hot-swap slots…

@scottalanmiller said:

So the real question is this....

What makes 400K IOPS without RAID worth $600 - $800 when 300K IOPS with RAID is just $300 for this specific use case?

@scottalanmiller said:

And, it should be pointed out, that a $300 RAID 5 array here is likely safer (both in terms of continuous uptime as well as in terms of dataloss) than the PCIe SSD + the SAN replication. If it were me, and I had to choose between the RAID array and the async replication to an external SAN I'd take the SSD RAID 5 array because it is fully consistent, not just crash consistent.

Unfortunately, this is not my case: OEM SSD aren't supported with our RAID cards in the servers, and VMware can't do software RAID (apart from, well, sort of, uhm, VSAN).
IBM's SAS SSD are still incredibly expensive.

@scottalanmiller said:

@Francesco-Provino said:

We will do both async replication from SSD to SAN (direct attach Fibre Channel, already in place, our main storage pool as of today) and backup to a NAS unit (QNAP, big SATA drives).

So the failover to the SAN is risky in that data could be lost because it is only crash consistent and the filesystem and/or databases might be corrupted when attempting to use it.

What is the time of dataloss if you need to go to the QNAP to do a restore?

That's always true with async replication. QNAP is in the same building, connected with gigabit network. In my tests, I can retrieve the backup of our biggest VM in almost an hour and a half. Totally ok for us.

@scottalanmiller said:

@Francesco-Provino said:

As I said, I can restart the VMs almost immediately on the SAN (they are replicated, so ready to be restarted), or restore them, both from replication pool or from backup, to one of the other two servers local storage.

What is the manner of replication?

VMware Replication to the SAN, Veeam to the NAS.

@scottalanmiller said:

@Francesco-Provino said:

I know about It, but thanks to the replication I think we can live with that. We can have few hours of downtime without losing too much money.

How quickly does Intel do replacements? Intel is not an enterprise supplies like HP, Dell or Fujitsu.

As I said, I can restart the VMs almost immediately on the SAN (they are replicated, so ready to be restarted), or restore them, both from replication pool or from backup, to one of the other two servers local storage.

So, we can wait some days for Intel.

@scottalanmiller

@scottalanmiller said:

@Francesco-Provino said:

I know about the legendary reliability of FusionIO, but… I don't think we really need THAT much reliability, not at this price! Replicating every VM to SAN, I can power on the VMs of a failure node in almost no time directly from the SAN.

There are three ways to handle this replication:

• Full Synchronization replication
• Asynchronous replication
• Backup mechanisms

Of these you have these impacts or tradeoffs:

Full Sync: This is a form of network RAID 1. You will need to wait for the SAN to respond that it has written a copy of the data. While your read performance will be as fast as the Intel PCIe SSD can go, the writes will be as slow as the SAN can do. So while this is safe and allows for storage failover without dataloss or downtime, the impact to writes is enormous.

Async: Data is only crash consistent. You can have "nearly every byte" that you had before but data can and sometimes does corrupt. It cannot be tested as corruption only happens some of the time and typically happens under load. So there is a risk that your SAN would be corrupted and useless in the event of the PCIe SSD failing.

Backup/Restore: Needs quiescence to be safe which inflicts a performance penalty on its own. In the event of a PCIe SSD failure you are doing a DR scenario and facing some dataloss.

So there are options, each with different caveats. It would depend on what needs your business has as to which would make sense for you.

Thanks for the clarification on replication, I really appreciate it.
We will do both async replication from SSD to SAN (direct attach Fibre Channel, already in place, our main storage pool as of today) and backup to a NAS unit (QNAP, big SATA drives).

@scottalanmiller said:

@Francesco-Provino said:

@scottalanmiller So Intel 5-years warranty has no such value in this case? I'll be happy to replace them every 3-4 years…

Warranties have little value when you are talking about your data and uptime. A warranty is to guarantee that you have equipment for the duration, not that the things that you store on that equipment continue to exist. If we are talking a desktop on which no critical data is stored and you have a spare desktop to use until Intel replaces the SSD, sure, the warranty has value. If we are talking about a server holding your critical data the warranty presumably has almost no value.

When the PCIe SSD fails you will need to order the warranty replacement. What is the replacement terms - four hours, six hours, next business day, two weeks? Do you have to return the failed one first and wait for them to test it? Remember this is a complete storage system not just one drive in a RAID array. When HP or Dell do warranty replacement of a drive there is no downtime or dataloss. When Intel does a replacement of these drives, you are without storage for some amount of time and once replaced, the data from the old SSD is gone.

I know about It, but thanks to the replication I think we can live with that. We can have few hours of downtime without losing too much money.
We mainly do VDI and database stuff… it's not that we require such great IOPS count, but… what are the alternatives? Buy IBM spindles in 2015, at an higher price of the SSD? Double the price for 1/100 IOPS? Does it really makes sense?

@scottalanmiller said:

@Francesco-Provino said:

We need to move to local storage, and it seems to me that this is the most convenient approach; but anyway, I was trying to fetch some information about reliability…

Traditional enterprise boards like FusionIO have very good reliability track records. Intel is new to the game and has a good reputation in the SSD space and a bad one in "non-drive storage space." Put the two together and this would be a rather unknown scenario with them.

I know about the legendary reliability of FusionIO, but… I don't think we really need THAT much reliability, not at this price! Replicating every VM to SAN, I can power on the VMs of a failure node in almost no time directly from the SAN.

@scottalanmiller said:

@Francesco-Provino said:

@Reid-Cooper As of today, IOPS/€-wise the NVMe PCIe are actually way more cheaper than SAS SSD!

By a combination of removing the SATA bottleneck, but also by skipping the RAID.

Exactly, I think this is definitely a win-win approach.

@scottalanmiller So Intel 5-years warranty has no such value in this case? I'll be happy to replace them every 3-4 years…

@Reid-Cooper As of today, IOPS/€-wise the NVMe PCIe are actually way more cheaper than SAS SSD!

The price of a set of SAS SSD from IBM (the only ones supported by the RAID controller that I have in my servers!) that match IOPS and capacity of an Intel (or other brands, of course) PCIe SSD is roughly three times…

We need to move to local storage, and it seems to me that this is the most convenient approach; but anyway, I was trying to fetch some information about reliability…

@scottalanmiller Specifically, I'm talking about Intel p3500 or p3600 in any server VS IBM FC SAN DS3500 direct attached (so, DAS in truth) with 12x15k spindles. Or, considering HDD local storage, a m5110 RAID card on each server.

Hello, I'm about to move one of the infrastructure I administer from IPOD (one SAN with 12 spindles) to local PCIe SSD replicated into SAN. I just want to hear some opinion about single PCIe SSD reliability VS a spindles array…

Or, better, should the PCIe card be as reliable as an hardware RAID card? NAND wearing aside, of course.

Hi everybody from Sicily!