The FOSDEM folks have processed and released the video on my “Live Migration of Virtual Machines From The Bottom Up” talk. Available at this location.
Yesterday was the first day at devconf.cz 2015. It’s my first devconf.cz, and I’m impressed by the large turnout and the perfect management of the event by the organizers.
Yesterday was also the day I presented my talk on live migration of QEMU/KVM VMs. The slides are here. There was also live video broadcast; and the recording is at this link. You’ll have to select the E104 section to view my talk. Also, that selection process needs flash. Unfortunate. I’ll check if there’s a direct link to my part of the talk.
Update: The direct link to the talk is here, thanks Donovan.
I will also do a follow-up post with a textual version of the talk in a few days’ time.
The Linux Plumbers Conf wiki seems to have made the discussion notes for the 2012 conf read-only as well as visible only to people who have logged in. I suspect this is due to the spam problem, but I’ll put those notes here so that they’re available without needing a login. The source is here.
These are the notes I took during the virtualization microconference at the 2012 Linux Plumbers Conference.
Several applications need random numbers for correct and secure operation. When ssh-server gets installed on a system, public and private key paris are generated. Random numbers are needed for this operation. Same with creating a GPG key pair. Initial TCP sequence numbers are randomized. Process PIDs are randomized. Without such randomization, we’d get a predictable set of TCP sequence numbers or PIDs, making it easy for attackers to break into servers or desktops.
On a system without any special hardware, Linux seeds its entropy pool from sources like keyboard and mouse input, disk IO, network IO, and any other sources whose kernel modules indicate they are capable of adding to the kernel’s entropy pool (i.e .the interrupts they receive are from sufficiently non-deterministic sources). For servers, keyboard and mouse inputs are rare (most don’t even have a keyboard / mouse connected). This makes getting true random numbers difficult: applications requesting random numbers from /dev/random have to wait for indefinite periods to get the randomness they desire (like creating ssh keys, typically during firstboot.).
I’ve been using the Fedora 18 pre-release for a couple of months now, and am generally happy with how it works. I filed quite a few bugs, some got resolved, some not. Here’s a list of things that don’t work as they used to in the past, with workarounds so they may help others:
Updating a Fedora 16 guest to a Fedora 17 guest via preupgrade gave me the ‘Oh no, something has gone wrong!’ screen at the GDM login screen. It’s quite frustrating to see that screen because you can’t switch to a virtual terminal for troubleshooting, or even reboot or shutdown.
To send the key sequence Ctrl+Alt+F2 to the guest to switch to a virtual terminal, use the qemu monitor by pressing
and use sendkey to send the key sequence:
(qemu) sendkey ctrl-alt-f2
Then go back to the guest window by issuing
After logging in as root, I poked in the gdm log files in /var/log/gdm/ and saw the fprint daemon was causing some errors. Removing the fprintd package fixed this, but this is just a workaround, not a solution:
yum remove fprintd
My second talk at FUDCon Pune was on Virtualization (slides) on day 2. While I had registered the talk well in advance, I wasn’t quite sure what really to talk about: should I talk about the basics of virtualization? Should I talk about what’s latest (coming up in Fedora 16)? Should I talk about how KVM works in detail? My first talk on git had gone well, and as expected for this FUDCon, majority of the participants were students. Expecting a similar student-heavy audience for the 2nd talk as well, I decided on discussing the basics of the Linux Virt Stack. Kashyap had a session lined up after me on libvirt, so I thought I could give an overview of virt-manager, libvirt, QEMU and Linux (KVM).
And since my registered talk title was ‘Latest in Linux Virtualization’, I did leave a few slides on upcoming enhancements in Fedora 16 (mostly concentrating on the QEMU side of things) at the end of the slide deck, to cover those things if I had time left.
As with the previous git talk, I didn’t get around to making the slides and deciding on the flow of the talk till the night before the day of the talk, and that left me with much less sleep than normal. The video for the talk is available online; I haven’t seen it myself, but if you do, you’ll find I was almost sleep-talking through the session.
To make it interactive as well as keep me awake, I asked the audience to stop me and ask questions any time during the talk. What was funny about that was the talk was also being live streamed, and the audio signal for the live streaming was carried via one mic and the audio stream for the audience as well as the recorded talk was on a different mic. So even though the audience questions were taken on the audience mic, I had to repeat the questions for the people who were catching the talk live.
I got some feedback later from a few people — I missed to introduce myself, and I should have put some performance graphs in the slides, as almost all users would be interested in KVM performance vs other hypervisors. Both good points. The performance slides I hadn’t thought about earlier, I’ll try to incorporate some such graphs in future presentations. Interestingly, I hadn’t also thought of introducing myself. Previously, I was used to someone else introducing me and then me picking up from there. At the FUDCon, we (the organisers) missed on getting speaker bios, and didn’t have volunteers introduce each speaker before their sessions. So no matter which way I look at it, I take the blame as speaker and organiser for not having done this.
There was some time before my session to start and there were a few people in the auditorium (the room where the talk was to be held), so Kashyap thought of playing some Fedora / FOSS / Red Hat videos. (People generally like the Truth Happens video, and that one was played as well.) These, and many more are available on the Red Hat Videos channel on YouTube. There was also some time between my session and Kashyap’s (to allow for people to move around, take a break, etc.), so we played the F16 release video that Jared gave us.
Overall, I think the talk went quite well (though I may have just dreamed that). I tried to stay awake for Kashyap’s session on libvirt to answer any questions directed my way; I know I did answer a couple of them, so I must have managed to stay up.
Videos from the KVM Forum 2011 are being uploaded at http://www.youtube.com/playlist?list=PL7C0F52E2227156B3
Guest and Host communication should be a simple affair — the venerable TCP/IP sockets should be the first answer to any remote communication. However, it’s not so simple once some special virtualisation-related constraints are added to the mix:
- the guest and host are different machines, managed differently
- the guest administrator and the host administrator may be different people
- the guest administrator might inadvertently block IP-based communication channels to the host via firewall rules, rendering the TCP/IP-based communication channels unusable
The last point needs some elaboration: system administrators want to be really conservative in what they “open” to the outside world. In this sense, the guest and host administrators are actively hostile to each other. Also, rightly, neither should trust each other, given that a lot of the data stored in operating systems are now stored within clouds and any leak of the data could prove disastrous to the administrators and their employers.
So what’s really needed is a special communication channel between guests and hosts that are not susceptible to being blocked out by guests or hosts as well as being a very special-purpose low-bandwidth channel that doesn’t look to re-implement TCP/IP. Some other requirements are mentioned on this page.
After several iterations, we settled on one particular implementation: virtio-serial. The virtio-serial infrastructure rides on top of virtio, a generic para-virtual bus that enables exposing custom devices to guests. virtio devices are abstracted enough so that guest drivers need not know what kind of bus they’re actually riding on: they are PCI devices on x86 and native devices on s390 under the hood. What this means is the same guest driver can be used to communicate with a virtio-serial device under x86 as well as s390. Behind the scenes, the virtio layer, depending on the guest architecture type, works with the host virtio-pci device or virtio-s390 device.
The host device is coded in qemu. One host virtio-serial device is capable of hosting multiple channels or ports on the same device. The number of ports that can ride on top of a virtio-serial device is currently arbitrarily limited to 31, but one device can very well support 2^31 ports. The device is available since upstream qemu release 0.13 as well as in Fedora from release 13 onwards.
The guest driver is written for Linux and Windows guests. The API exposed includes open, read, write, poll, close calls. For the Linux guest, ports can be opened in blocking as well as non-blocking modes. The driver is included upstream from Linux kernel version 2.6.35. Kernel 2.6.37 will also have asynchronous IO support — ie, SIGIO will be delivered to interested userspace apps whenever the host-side connection is established or closed, or when a port gets hot-unplugged.
Using the ports is simple: when using qemu from the command line directly, add:
-chardev socket,path=/tmp/port0,server,nowait,id=port0-char -device virtio-serial -device virtserialport,id=port1,name=org.fedoraproject.port.0,chardev=port0-char
There is sample C code for the guest as well as sample python code from the test suites. The original test suite, written to verify the functionality of the user-kernel interface, will in the near future be moved to autotest, enabling faster addition of more tests and tests that not just check for correctness, but also regressions and bugs.
virtio-serial is already in use by the Matahari, Spice, libguestfs and Anaconda projects. I’ll briefly mention how Anaconda is going to use virtio-serial: starting Fedora 14, guest installs of Fedora will automatically send Anaconda logs to the host if a virtio-serial port with the name of ‘org.fedoraproject.anaconda.log.0‘ is found. virt-install is modified to create such a virtio-serial port. This means debugging early anaconda output will be easier with the logs available on the host (and not worrying about guest file system corruptions during install or network drivers not available before a crash).
Further use: There are many more uses of virtio-serial, which should be pretty easy to code:
- shutting down or suspending VMs when a host is shut down
- clipboard copy/paste between hosts and guests (this is under progress by the Spice team)
- lock a desktop session in the guest when a vnc/spice connection is closed
- fetch cpu/memory/power usage rates at regular intervals for monitoring
Navin gives a small demo of RHEV-M, the management platform for the Red Hat Enterprise Virtualization product: