Playing with libvirt

I've been playing with libvirt in the last two days. My motivation was that I saw:

1. plain kvm is not that fast
2. there is this virtio framework which aims to make it faster
3. using virtio directly is hard, better not reinventing the wheel and just try libvirt

So if you want to try it yourself, you need to pacman -S libvirt, service libvirtd start. Check /var/log/syslog for error messages, in case it can't find some commands (bridge-utils, dnsmasq, etc.), install them.

If you want to create a new virtual machine, you need pacman -S virtinst. Then you can use something like:

virt-install --name=syncpkgcd-helicon --arch=x86_64 --vcpus=1 --ram=1024 --os-type=linux --os-variant=virtio26 --connect=qemu:///system --network network=default --cdrom=/virt/iso/frugalware-1.3-i686-net.iso --disk path=/virt/syncpkgcd/syncpkgcd.img,size=40 --accelerate --vnc --noautoconsole --keymap=us

See man virt-isntall for more info about the meaning of the switches. One trick: kvm can't handle the gfxmenu in our grub on the install cd, so you need to disable that. One way to do it is to use vim -b frugalware-1.3-i686-net.iso, search for "menu /boot", and change "gfx" to "#fx". Then you can run virt-install. It'll start the install in the background. The easiest way (for me) to connect to the console is to create an ssh tunnel to the server running the kvm machine (for example ssh -L 5900:localhost:5900 server), then run krdc localhost:0 on my own machine.

Next trick is that you need kernel support to boot from virtio disks (/dev/vda), that's enabled in frugalware-current.git but there were no rebuild yet, so you need to build it yourself for now.

Once the installation finished, the machine will be shut down. You can use virsh to start it. Here is a great summary about virsh subcommands. Something not to forget: if you want to automatically start the machine after the host booted, use virsh autostart $name.

(You can also do a service libvirt-guests add, that way all guests will be automatically suspended/resumed on shutdown/boot. Just to be clear: if you want the "resume if it was a proper shutdown, start if there was a power cut" feature: you need both init scripts because libvirtd will always start/resume autostart guests, but it will never suspend them on shutdown.)

If grub hangs after reboot, try creating a small (I used 32MB) separate /boot partition first, that should fix the issue.

virsh shutdown $machine won't work if you don't install acpid and enable it as well, so it's a good idea to do so.

The last trick: if you want to use the virt-manager gui from your local machine to manage a remote host, it isn't trivial to do so. First, it would try to login via ssh as root, and that's disabled by default. (And why would you enable it?). Also, even if you allow that, it would try to use nc -U which is supported by the openbsd nc only, the gnu nc does not have such a switch. Instead, we can use socat to connect to unix sockets. So I created a wrapper script under ~/bin with the following contents:

#!/bin/sh
if [ "$1" == "-U" ]; then
        sudo socat - unix-client:$2
else
        /usr/bin/netcat $@
fi

Don't forget to add PATH=what_you_want line to ~/.ssh/environment, and include the full path of ~/bin there. This way you can keep using gnu nc and you can also login as a user to manage your machines, as long as you set up nopasswd sudo for your user for socat.

Finally a mini-benchmark:

The SBU of yugo (i686 buildserver, a bit old HW) is 645 seconds, SBU of an i686 guest on helicon (c2q machine with 8g ram) is 264, SBU of the host build on helicon is 76.

I would call this usable (not right now, but once we sort out the kernel part). Finally something that is free software and is comparable to VMware ESX. :)

Update: I recently did the same install with 1.4pre1 (which installs -current, so the kernel is 2.6.36), the list of tricks you still need:

• separate /boot (haven't tried without)
• edit iso image to get rid of gfxboot (tried, still fails without)
• some problem with grub, you need to manually add "(hd0) /dev/vda" to /boot/grub/device.map, then run grub-install /dev/vda

So no more custom kernel, boot from virtio works by default! :)