Friday, October 28, 2016

KVM for IBM z Systems v1.1.2 released

KVM for IBM z Systems v1.1.2 is out today! See here for the respective blog post on Mainframe Insights, and the following pages for the formal announcements: US, Canada, Asia-Pacific, Japan, Latin America and Europe.

It ships with QEMU v2.6 and libvirt v1.3.3. Here is a list of highlights from a KVM on z perspective:
  • Enhanced SCA support
    Exploit up to 248 CPUs per KVM guest
  • SIE capability
    Expose SIE availability in /proc/cpuinfo as follows:

       $ cat /proc/cpuinfo | grep features
       features        : esan3 zarch stfle msa ldisp eimm dfp edat etf3eh
                         highgprs te vx sie
  • STP host toleration
    Previously, Server Time Protocol (STP) had to be turned off in the host. In case of time differences, the TOD clock is adjusted in a smooth manner, avoiding jumps.
  • Improved removable media support
    Allow to boot KVM guests seamlessly from ISO9660 media.
  • OASIS VIRTIO v1.0 support
    Support OASIS OASIS standard for virtio devices.
  • CPU hotplugging
    Allow to add CPUs dynamically to a running guest
  • STHYI support: Added the z/VM-defined Store Hypervisor Information instruction (STHYI) for KVM guests. See also here.

Thursday, October 6, 2016

Linux kernel 4.8 released

Linux kernel 4.8 (available here) has been released, here are the highlights in support of KVM on z:

Nested Virtualization
This feature allows to start further KVM hosts within KVM guests, also called second level virtualization. As a prerequisite, it requires a recent, post-2.7 version of QEMU including the s390 CPU models (assumed to be included in the forthcoming QEMU 2.8 release).
Nested virtualization is currently turned off per default, and has to be enabled when loading the kvm module:

   [root ~]# modprobe kvm nested=1

or by appending kvm.nested=1 to the kernel command line.
When starting QEMU, make sure to chose the right machine ("s390-ccw-virtio-2.8" or higher) in the domain xml's os element for the guest:

      <type arch='s390x' machine='s390-ccw-virtio-2.8'>hvm</type>
      <boot dev='hd'/>

Finally, verify in a KVM guest that hosting further KVM guests is possible, as indicated by flag "sie" in /proc/cpuinfo as follows:

   [root ~]# cat /proc/cpuinfo | grep features
   features        : esan3 zarch stfle msa ldisp eimm dfp
edat etf3eh
                     highgprs te vx sie

STHYI Instruction available for KVM Guests
Linux kernel 4.8 saw the inclusion of an implementation of the STHYI (Store Hypervisor Information) instruction. Using Linux kernel 4.8 for the host will make the STHYI instruction available to all KVM guests of that host.
Previously available on z/VM only, this instruction provides detailed information on CPU resources on various levels (machine, LPAR, hypervisor, guest, etc.).

Use qclib to access the information. Also see this blog post for further details regarding qclib and KVM on z.

Thursday, September 15, 2016

QEMU v2.7 released

QEMU v2.7 is out! Here is a list of the highlights for KVM on z:
  • Support for PCI passthrough devices is now fully functional for the first time. While we consider support as experimental, since there is still work to be done to utilize hardware acceleration facilities as well as further testing pending, attaching PCI devices directly to guests is now possible. Also note that while hot-plugging PCI devices is supported, hot-unplugging is not. Further limitations, e.g. for live guest migration, apply for now.
    We strictly recommend to use libvirt for all configuring, including PCI devices. Therefore, a step-by-step description of how to attach a PCI device to a KVM guest on z Systems will follow when the respective code becomes available in one of the next libvirt releases.

  • Runtime instrumentation in KVM guests: IBM z13 and LinuxONE machines introduced a new hardware facility which allows to collect sampling data on code execution in CPUs at runtime. This data can be used by just-in-time compilers, like the one provided by IBM Java, to improve the generated code for better performance.
  • IPL from any subchannel set: Prior to QEMU v2.7, KVM guests could only IPL/boot from devices with a subchannel set ID of 0. Once IPL'd, they could attach disks in any subchannel set just fine, but the IPL/boot from devices in subchannel sets 1-3 was not possible. With the addition of this feature, KVM guests can IPL from devices in all 4 subchannel sets.
    Here is a sample domain definition indicating the change:

      <disk type=’file’ device=’disk’>
    <driver name=’qemu’ type=’qcow2’/>
      <source file=’/var/lib/libvirt/images/guest.qcow2’/>
        <target dev=’vda’ bus=’virtio’/>
        <address type=’ccw’ cssid=’0xab’ ssid=’0x2’

Wednesday, September 7, 2016

Open Positions in Development

Recently graduated (or about to do so) and looking for a job in KVM or Linux development on z Systems?
Then go to this page and search for
  • 59794BR for a position in KVM on z development ("Entwickler/in für Linux Virtualisierung auf IBM z Systems"), or
  • 59784BR for a position in Linux on z development ("Linux Entwickler für IBM z Systems (m/f)")

Friday, July 29, 2016

Linux kernel 4.7 released

In Linux kernel 4.7 (available here), we improved usability of the kvm_stat tool for z Systems, introducing (among others) new option -p to filter by process ID.

Using kvm_stat, one can monitor e.g. the number of SIE exits or specific diags to investigate performance issues:
  kvm statistics - summary

   Event                                        Total Current
   kvm_s390_sie_enter                          621113      96
   kvm_s390_sie_exit                           621113      96
   kvm_vcpu_wakeup                             231054      69
   kvm_s390_intercept_instruction              187720      24
   kvm_halt_poll_ns                            131133      15
   kvm_s390_handle_diag                         64543      12
   kvm_userspace_exit                            1427      11
   kvm_s390_handle_sigp                        121746       1
   kvm_set_irq                                 154402       0
   kvm_s390_handle_sigp_pei                     18407       0
   kvm_s390_handle_stsi                            12       0
   kvm_s390_handle_stfl                             1       0
Use option -p to further investigate specific QEMU processes to identify the potential culprits:
  $ ./kvm_stat -p 4304 -1 | grep kvm_s390_sie
  kvm_s390_sie_enter                              3985      3985
  kvm_s390_sie_exit                               3985      3985
  kvm_s390_sie_fault                                 0         0

Wednesday, July 13, 2016

Webinar: KVM for z Systems Performance

Abstract: KVM for IBM z Systems performance has not been presented at conferences before. This session covers performance aspects of the KVM host, virtual devices, and considerations for guest definitions to reach certain performance levels. It will also give examples how to monitor host and guest.
Note: It is not intended to compare KVM for IBM z Systems to other hypervisors.
Speaker: Martin Kammerer, Manager Linux on z Systems Performance Evaluation, IBM Germany

Audience: IT Managers, IT Architects, System Administrators

Date: Wednesday, July 20, 2016
Time: 11:00 AM EST / New York, 4:00 PM UK, 5:00 PM CET / Germany
Duration: 75 minutes

Registration: You can register here. After registering, you will receive a confirmation email containing information about joining the webinar.

Replay & Archive
: All sessions are recorded. For the archive as well as a replay and handout of this session (available by July 25, 2016), as well as all previous webcasts please visit

Contact: Please direct any questions to Stephanie Gherghe at

Friday, July 8, 2016

KVM on z Documentation References

The amount of documentation provided in the IBM Knowledge Center can at times be overwhelming. Therefore, this constantly-updated blog entry will provide some general guidance.

The documentation for KVM on z (see overall entry point here) is grouped under these categories:
For white papers, Redbooks, live virtual classes, and more, check out the materials related to virtualization on z Systems available here.

In addition, here is a document listing the recommended limits when running KVM on z Systems. It is aimed at the KVM for IBM z product, but also provides a good starting point when running other flavors of KVM on z.

Finally, documentation and pointers on ICM (IBM Cloud Manager, IBM's OpenStack-based product) can be found here.