Kernel 3.3 and Xen 4.1.2 on openSuse 12.1

For over a year I used VMware Server 2 to run few virtual machines at work. But after every kernel update I had to look for a patch to build modules needed by VMS2. Eventually I got fed up with this procedure and started looking for a better alternative. Got XEN to run on Mandriva 2010, OS of choice at the time. Xen and kernel come as a team so no need to recompiling any thing, and it's a bit faster then VMS2. Sadly Mandriva 2011 didn't come with Xen full virtualization kernel, so I tried openSuse 12.1, after not so good experience with Fedora 16.
OpenSuse 12.1 comes with kernel 3.1.0 and xen 4.1.2. After updating to kernel 3.1.9, me and other users of virtual machines on my office server, noticed that net connection tends to break, for example when some1 starts his machine.

To allow every user to start VMs manually, I've added this to /etc/sudoers

%users  ALL=(ALL)         NOPASSWD: /usr/local/bin/xmst.sh

.. and /usr/local/bin/xmst.sh looks like this

#!/bin/bash

export LD_LIBRARY_PATH=/xen/lib64:/xen/lib
export LD_RUN_PATH=/xen/lib64:/xen/lib
export PYTHONPATH=/xen/lib64/python2.7/site-packages

/xen/sbin/xm start $1

I've renamed original xen init scripts

cd /etc/init.d/
rename -v xen z-xen xen*

1. Building kernel

# as root
cd /usr/src
curl -O http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.3.tar.bz2
tar xjf linux-3.3.tar.bz2
cd linux-3.3

a) configure
You can start with

make  allyesconfig

and just uncheck (in make menuconfig) what you don't need.

I've used .config from rpm of kernel-xen-3.1.9 as a base of my configuration.

DO NOT USE LZMA/XZ COMPRESSION FOR KERNEL/INITRD - Xen only supports gzip compression.

make menuconfig
make
make modules_install
make install

I've also copied the bare kernel image, just in case ;].

cp /usr/src/linux-3.3/vmlinux /boot/vmlinux-3.3

Building Xen

env PREFIX=/xen  make dist
make install

Now there are a few files to change. In /etc/init.d : xen-watchdog, xencommons, xend, xendomains (this one I don't use)

Here's what to add after INIT INFO block:

export LD_LIBRARY_PATH=/xen/lib64:/xen/lib
export LD_RUN_PATH=/xen/lib64:/xen/lib
export PYTHONPATH=/xen/lib64/python2.7/site-packages
export PATH=/xen/bin/xen/sbin:$PATH

I've prepended PATH with my build bins because I don't want to use those installed from rpm.

And added something like this /boot/grub/menu.lst

title Xen - openSUSE 12.1 - 3.3.0
    root (hd0,0)
    kernel /boot/xen-4.1.2.gz vga=mode-0x31a dom0_mem=4488M noreboot
    module /boot/vmlinuz-3.3.0-xen root=/dev/sda1 showopts console=tty7 xencons=tty8
    module /boot/initrd-3.3.0-xen

I've copied vms config.sxp to /xen/vm and set path in /xen/etc/xen/xend-config.sxp

(xend-domains-path /xen/vm/)

You can see that process running my vm has opened vnc access.

ps aux | grep xen

/xen/lib/xen/bin/qemu-dm -d 1 -domain-name dom1 -videoram 4 -k pl-pl -vnc 0.0.0.0:2,password -parallel none -vcpus 1 -vcpu_avail 0x1L -boot c -localtime -serial pty -acpi -usb -usbdevice tablet -net nic,vlan=1,macaddr=00:16:3e:3c:48:65,model=rtl8139 -net tap,vlan=1,ifname=tap1.0,bridge=virbr0 -M xenfv

I've added paths to my xen libs in /etc/ld.so.conf.d/xen.conf like so

/xen/lib
/xen/lib64

After running ldconfig I could probably comment-out those exports in /etc/init.d/my-xen*, but I did it mostly so I could run xentop without typing freaking long paths like this

env LD_LIBRARY_PATH=/xen/lib64:/xen/lib  PYTHONPATH=/xen/lib64/python2.7/site-packages  /xen/sbin/xentop

Since openSuse 12.1 includes custom confs for ld first, my libs will be used before those installed from rpm - in /usr/lib64.

ldd xenconsoled xenstored xenwatchdogd
xenconsoled:
        linux-vdso.so.1 =>  (0x00007fff3c288000)
        libxenctrl.so.4.0 => /xen/lib64/libxenctrl.so.4.0 (0x00007f6a35c82000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f6a35a7e000)
        libxenstore.so.3.0 => /xen/lib64/libxenstore.so.3.0 (0x00007f6a35874000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f6a35671000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f6a35469000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f6a350d9000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6a34ebc000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6a35ea5000)
xenstored:
        linux-vdso.so.1 =>  (0x00007fff64d85000)
        libxenctrl.so.4.0 => /xen/lib64/libxenctrl.so.4.0 (0x00007fe712f30000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fe712d2c000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fe71299c000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe71277f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe713153000)
xenwatchdogd:
        linux-vdso.so.1 =>  (0x00007fff6a149000)
        libxenctrl.so.4.0 => /xen/lib64/libxenctrl.so.4.0 (0x00007f5b01bb5000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f5b019b1000)
        libxenguest.so.4.0 => /xen/lib64/libxenguest.so.4.0 (0x00007f5b0178b000)
        libxenstore.so.3.0 => /xen/lib64/libxenstore.so.3.0 (0x00007f5b01581000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f5b011f1000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5b00fd4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5b01dd8000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f5b00dbc000)

After checking that everything works, I've added system wide path to my xen bins
-in /etc/profile.local

export PATH=/xen/bin:/xen/sbin:$PATH

Here's a example how to use xl - the replacement for xm

xl vcpu-pin dom1 0 0
xl vcpu-pin dom1 1 1
xl vcpu-pin dom2 0 2
xl vcpu-pin dom2 1 3

- pins physical cpu (cores) 0 & 1 to dom1s virtual cpu 0 & 1, and physical cpu 2 and 3 to dom2 virtual cpu 0 & 1.

Xen 4.2 is due this or next month, hope it will be as easy to install ;].

UPDATE

Here's .config from my xen kernel 3.3.
Weird thing is - I've tried same .config with kernel 3.3.1 when it came out - can't get the damned thing to work.
Kernel on it's on boot fine, but with xen - ends with blank screen and reboots (removed noreboot from grub) - kernel 3.3 & xen works fine, so I guess I'll take a crack at next kernel version (3.3.2 / 3.4).


UPDATE 2
First thing I did at work on monday morning was compile kernel 3.3.2 ... and Xen works again, woo hoo :D