[GeoffM]

First off Excellent article, no need for negitive comments here, you addressed a needed niche, we're right in the middle of testing our VC3 Infrastructure deployment and before I order my c-Class Blade and iSCSI SAN I was looking for a way to test the full functions of VM. Thank you!

Now if only you could find a way to turn a WD My Book (Networked attached) in to a sedo iSCSI I'd really have it made

[jasonboche]

You could connect your book to openfiler or a Windows iSCSI machine to be used as storage, but performance wouldn't be real good fo running more than 1 or 2 VMs concurrently.

[Matz]

Hi.

-> jasonboche
OpenFiler, do you mean you not use more than 2 VM "running" on the of iscsi?
Why, what are the bottleneck?

Have you tried to use VMotion, did it work?


-> GeoffM
There are also FreeNas and sw based StarWind, both are good and easy to set up.

Thanks.
Matz

[cmarkley]

I tried FreeNAS and OpenFiler and didn't have much luck. I friend turned me onto DSS and it rocks. We are running Vmotion and HA w/o any issue. It runs on a USB thumb drive and they have a 1TB version for free that we use.

http://www.open-e.com/dss_lite/data_...ow=description

[jasonboche]

Originally Posted by Matz

Hi.

-> jasonboche
OpenFiler, do you mean you not use more than 2 VM "running" on the of iscsi?
Why, what are the bottleneck?

Have you tried to use VMotion, did it work?


-> GeoffM
There are also FreeNas and sw based StarWind, both are good and easy to set up.

Thanks.
Matz

The limitation isn't iSCSI, it's the disk you'd be runnig it on. Because you'd be running VMs on a single IDE or SATA drive in a USB "book" drive, disk I/O would start to show serious contention and bottleneck probably after the 2nd VM.

VMotion works just great on iSCSI. Been using it and Openfiler for close to 2 years.

[Matz]

Hi

Thanks for great input.

-> jasonboche
Put together some old hw.
Installed OpenFiler on a HP DL380 with 2x36GB for sys and 4x 300GB for the data.
1xGbit nic.

1. So you mean, you like OpenFiler and have used it successfully to use VMotion between 2 ESX (3,5) server(s)?

2. Any tweaks adjustment you did have to do?

3. What about performance?

*******************************
I have also installed DSS-Lite to see if that is a great iScsi application
Have you or anyone else tesed it?
What about the $$ version - Open-E Data Storage Server / Open-E iSCSI-R3 ?

*******************************

Thanks again for all input.
Matz

[jasonboche]

Originally Posted by Matz

Hi

Thanks for great input.

-> jasonboche
Put together some old hw.
Installed OpenFiler on a HP DL380 with 2x36GB for sys and 4x 300GB for the data.
1xGbit nic.

1. So you mean, you like OpenFiler and have used it successfully to use VMotion between 2 ESX (3,5) server(s)?

I have been using Openfiler as an iSCSI target for ESX 3.0.x and 3.5.x for the past 2 years which allows me to use and test all the enterprise features in virtual infrastructure such as VMotion, DRS, HA, DPM (experimental), etc. Shared storage really does provide a lot of flexibility for maintenance. Once you go shared storage, you'll never want to go back.

Originally Posted by Matz

2. Any tweaks adjustment you did have to do?

Well there's the Openfiler setup which is pretty straight forward. Then there's the configuration in Openfiler to present the disk(s) to ESX. It is pretty simple and can be done relatively quickly. Openfiler is a very fast build. As far as any specific settings or hardware you might want, I"ll speak to them below.

Originally Posted by Matz

3. What about performance?

Openfiler will install on a very wide variety of x86 and x64 hardware (if you have x64, make sure you are using the x64 compiled version of Openfiler!). As such, performance can vary greatly depending on your hardware setup. You can go as cheap as you want, recycling old hardware and minimum disk spindles which will result in a shared storage solution, but it may not perform the best. Alternatively, you could spend some money on ebay or at your local electronics store and make subtle improvements in your Openfiler hardware which will enhance the performance of your box resulting in better performing VMs and more capacity for more running VMs.

Let's go over a few of the key bottlenecks you want to address with iSCSI.

1. iSCSI is disk over network. Naturally for your iSCSI box you will want a Gigabit network card which your iSCSI traffic will traverse. A big bottleneck here would be going with a Fast Ethernet 10/100 NIC (or wireless!!), or an improperly configured Gb NIC. Naturally, you will want the other end of that Gb cable plugged into a Gb switch that supports Gb networking. Taking it a step further, you will want your ESX NICs to be Gb and also plugged into a Gb switch! Hopefully you are getting the point that sticking a 10/100 NIC anywhere in the mix will result in a less than stellar performing iSCSI datastore. While we're on the subject of NICs, I want to point out that there does exist a NIC bonding driver which rPath Openfiler Linux will support which means you can multihome your Openfiler box and bond the NICs to aggregate traffic throughput resulting in 2Gb+ speeds. This is a highly complex tweak and there is some fine print and caveats that you will have to take a look at but the option is there for you. I can provide you with the documentation on this if you'd like to explore it further.

2. Processor(s): You will find out by running TOP on your Openfiler box that the processor will become loaded when throwing a lot of iSCSI traffic at it. That said, the faster the processor, the better. Openfiler supports SMP so if you can find a multi proc box, definitely go that route. My Openfiler box is currently a 4 way P4 2.x GHz.

3. RAM: Doesn't appear to come into play much so you can rest easy in this category. My box has 9.5GB of RAM of which I don't think I'm even getting close to using a gig of it..

4. Array controller, disk spindle count, and disk spindle speed. I saved the best for last. By far the greatest improvements (and most expensive) you can make to your iSCSI box are increasing your disk spindle count to the most possible (or affordable) and going with an array controller that will be able to keep up with the number of disk spindles and the I/O reads/writes. My server has a Compaq SMART Array 6400 array controller and 12x 146GB 10KRPM hot pluggable SCSI drives. While I can't afford 15KRPM SCSI drives, that would make another great improvement in my setup. Maybe someday.

That's all for now, I'm about to get off the bus here. Good luck!!

I came back to tack on a concluding statement now that I have a little more time. I wanted to point out that even the fastest of iSCSI hardware or the average fiber channel SAN will flounder a bit when overloading it with disk I/O requests from the ESX hosts. Examples usually include doing something with multiple VMs at the same time which are disk I/O intensive. For instance, powering on several VMs at the same time. Installing a large service pack on several VMs at the same time, etc. This is where you'll see your Gb NIC shine, but your disks will be working overtime trying to keep up with all the requests. If you run TOP, you'll also likely see the processors on your Openfiler box are buried. Another thing to remember is that if you're using the swISCSI initiator in ESX, there's some overhead in that guy too on the ESX side. You'll see this by running ESXTOP on your ESX hosts when your iSCSI box is working hard. Let me clarify - it's not that your VMs will start failing as the Openfiler box works hard, they will just be slower as it churns through the disk queues. Any shared storage solution has its limitations and assuming you cover all areas, your bottlenecks will usually be disk I/O so focus on disk I/O throughput when building an iSCSI box for ESX.

Originally Posted by Matz

*******************************
I have also installed DSS-Lite to see if that is a great iScsi application
Have you or anyone else tesed it?
What about the $$ version - Open-E Data Storage Server / Open-E iSCSI-R3 ?

*******************************

Thanks again for all input.
Matz

[GeoffM]

Originally Posted by Matz

Hi.

-> jasonboche
OpenFiler, do you mean you not use more than 2 VM "running" on the of iscsi?
Why, what are the bottleneck?

Have you tried to use VMotion, did it work?


-> GeoffM
There are also FreeNas and sw based StarWind, both are good and easy to set up.

Thanks.
Matz

Thanks Matz I'll look over both (along with DSS) very througly - I provide some SMB consaulting as well and got to thinking this would be a great cost point solution when combined with a Western Digital Networked MyWorlBook (Or simular) they have a 1 TB and a redundent 1TB (or 2 TB's if not used in redundentcy mode) and while they have a 1 gbs Network interface I do understand it's not a true iSCSI with I/O and other performance capabilities.

[sven.jambor]

Hi,

Just a word of caution.

I was surprised to read the article and see that Openfiler was used in this setup in combination with ESX's Enterprise features. I therefore assume that it's connected to more than one ESX host and the hosts are sharing the same iScsi targets.

I had a similar setup about a year ago - three ESX hosts and a 4TB shared target on Openfiler. The problem was that ESX would contunally get corrupt vmfs volumes, volume sizes would literally fluctuate during use, VM's woud disappear etc. So I ended up calling VMWare support (which did a great job, btw) - only to find out that Openfiler is not supported as a iScsi solution for the ESX's software redirector.

I said "yeah, so?" - but was informed that it's not only not supported - but that, like with most other non-certified iScsi solutions, it actually lacks the ability to use a few SCSI calls - which ESX DOES use. Hence the problems I was running in to.

In the end I just upgraded the exisiting "real" SAN with more storage (yup, spending MAJOR money). Right now we're using the Openfiler machine as a dedicated backup target - with huge disks and perfect performance. So all is well in the end anyway

But: if anybody has any news about ESX and Openfiler working together WELL in a PRODUCTION environment I'd be happy to hear about it!

[jasonboche]

I have been using Openfiler in my home production environment for 2 years or more with 3 ESX hosts. No problems.

Not many are going to suggest using a free opensource product like Openfiler for your work production environment. Spend a little money on a shared storage solution that you'll at least get support for if you have a problem.

Jas