iSCSI is fast (update: actually, it's not)

[update: iSCSI isn't so fast -- my benchmarks below are "buffer cache is fast". I went back to virtualbox disks instead, which are now even faster.]

After I built a ZFS server that uses 79W, I started looking around at other places I was using a bunch of power. And my Linux server (a few years old, P4) was using 150W+. What if I could turn it off and virtualize it?

I moved the whole thing to VirtualBox 3.0 with OpenSolaris as the host. This has worked reasonably well, aside from some odd hangs and a bunch of tuning you do to make the guest not eat all the host's CPU. (Hint: pass "divider=10" to your kernel. You don't need a new kernel.)

The glitches there are annoying enough that I recommend you mostly don't go the guest/host route. If you want to consolidate a bunch of servers, maybe you should try Xen or VMWare ESXi (which is sort of free now) instead.

And VirtualBox's performance overall is quite good (CPU, network).

The only problem is I put /home on NFS, and Solaris has all these rules, and they don't let you do async writes as much (like normal Linux NFS servers have let you do for years), and so compiling is very, very slow.

A small project of mine was taking about 2 minutes to compile over NFS. Same thing took 25 seconds on "local" Virtualbox make-believe disk. And yes I did "async" and "noatime" and all those things, and it didn't make a big difference.

So instead of adding another disk to Virtualbox, I decided to try out the iSCSI support in OpenSolaris. Would still get snapshots, would be exportable to a real physical box, and it's just cool to try.

After some help setting up the server (ahem, the "Target"), and setting up the client (the "Initiator"), things mostly worked, and I formatted with ext3.

23 seconds to do the compile.

Cool. I'm done with that stinky NFS volume.