Over the past two weeks I’ve been doing some work with virtualization in the context of Virtual Private Server setups – one physical host machine houses several “nodes” that do the actual processing.There are really two types of setups in this field: shared kernel and and separate kernel.
In a strictly shared kernel setup (like FreeBSD), the root kernel simple creates a new process tree for its nodes, and marks each process with both a process id (pid) AND it’s node id. This is obviously quite efficient, however it lacks the ability to do some of the handy things a separate kernel system provides.
In a separate kernel system (like Xen on linux, or even VMware), much more ram is neccesary since separate copies of all the binaries are loaded into memory. However, that means you can have different kernel version in different setups for testing, as well as better control over “virtual” hardware presented to those node kernels (I am keanly reminded of the networking restrictions FreeBSD jails have, as shared kernel nodes).
Most real operating systems support one setup or the other, however there are a few libraries out there that run on several OSs to allow for both options. Here’s what I’ve found.
Linux - After installing the Xen package, linux supports separate kernel nodes quite well. Despite some of the inherent inefficiencies of such a system, this setup is widely used and provided by companies such as slicehost and linode.
FreeBSD - Out of the box, FreeBSD includes support for Jails. This concept was written in 1995 to basically extend the functionality of a chroot environment. However, there are quite a few shortcomings: each jail is IDENTIFIED by its IP address, rather than a jail id or something of that sort. As such, each jail can have only ONE IPv4 address, and no IPv6. Further, there is no resource control to limit a jail to memory usage, cpu usage, or disk space (see end of paragraph for hacks). Also, a jail cannot do complicated firewalling or tunnels, but that is inherent to a shared kernel system so it can’t be blamed solely on FreeBSD’s implementation. It’s a shame out of the box jails suck so much, since BSD in general is a great system. There are a few patches to provide more jail control, however none are in the current source tree.
Solaris - Supporting 5 different types of virualization, Solaris really takes the cake. Everything works perfectly to any level of configuration. That said, Solaris x86 really is a scary beast. Even if you manage to get it installed, your hardware may still crash once a month when it throw a particular hook. If you’re using Sparc hardware however, this is most certainly the way to do things. (update 2011: the x86 builds are much more solid now)
In any event, I’m working on getting the patches to FreeBSD installed for jail control, as well as getting some nicer Sparc hardware to use Solaris.