Thursday, October 25, 2012

HowTo: Xen 4.1.3 Windows 8 HVM domU with Intel HD4000 VGA Passthrough on Debian Wheezy

Update 05/07/2013:
Despite the HowTo being close to 1 year it applies perfectly to this day. At the time Wheezy was Debian's Testing distribution and has since moved to Stable. Much of it can be used to setup other HVM domU such as Linux. 

Important notice: I've experienced some problems with xen-hypervisor-4.1-amd64 versions 4.1.4-3+deb7u1_amd64 and 4.1.4-3_amd64 which implement some security advisory patches . As a result to have sucessful passthrough I was forced to go back to the previous working version: 4.1.4-2_amd64 which is available at http://www.xen.org/downloads/XCP/debian/repo/amd64/xen-hypervisor-4.1-amd64_4.1.2-2.1_amd64.deb. Again, do notice that this package is missing some security patches available in more recent versions.
 
In this HowTo I'll present the steps required to install Xen 4.1.3 using the xm toolstack on a Debian Wheezy (kernel 3.2.0-3-amd64) dom0, create a Windows 8 HVM domU config and setup VGA/PCI Passthrough for the integrated GPU, USB 2.0 controller and audio.

This HowTo assumes that the reader is comfortable with Linux and Windows operating systems namely Debian GNU/Linux and Windows 7/8 as such it doesn't cover the operating systems installation.

For easier reference the procedure will broken down in the following steps:
1) Hardware requirements
2) Install Xen on Debian Wheezy
3) Configure networking
4) Configure Xen
5) Prioritise Xen boot
6) Create and install Windows 8 HVM domU
7) Assign devices for PCI Passthrough
8) Install GPLPV drivers
9) Advices and impressions

1) Hardware requirements

For PCI passthrough both motherboard and CPU must support VT-d also know as IOMMU IO virtualization.

The hardware used to write this HowTo setup is composed of:
  • Intel Core i7-3770 CPU,
  • Intel DQ77MK Motherboard,
  • 32GB GEIL DDR3 1600 MHz,
  • 200GB Maxtor SATA HDD,
  • Samsung SyncMaster 940BW Monitor.

It should be noted that VT-d and VT-x have been enabled in the motherboard and that the i7-3770 integrated GPU is the computer's sole GPU.

In addition to the above setup I've also used another networked computer so I could SSH into the dom0 and perform the steps identified bellow.

2) Install Xen on Debian Wheezy

The Xen hypervisor is provided by the xen-linux-system package:

  1. $ su
  2. # apt-get update
  3. # apt-get install xen-linux-system

3) Configure networking

There are several ways you can provide network access to domU guest domains, the most common being setting up a network bridge which I'll be covering.

To assign a static IP to the dom0 and define a network bridge named eth0, disable NetworkManager (if installed) and edit /etc/network/interfaces to contain a bridge:

  1. # /etc/init.d/network-manager stop
  2. # update-rc.d network-manager disable
  3. # aptitude install bridge-utils
  4. # vim /etc/network/interfaces auto lo br0 iface lo inet loopback allow-hotplug eth0 iface eth0 inet manual iface br0 inet static bridge_ports eth0 address 192.168.1.5 broadcast 192.168.1.255 netmask 255.255.255.0 gateway 192.168.1.254
  5. # vim /etc/resolv.conf domain my.domain.com nameserver 192.168.1.254

192.168.1.5 is the IP chosen to be assigned to the host, 192.168.1.255 and 255.255.255.0 are the typical broadcast and netmask values for a 192.168.1.x network and in my case the gateway is 192.168.1.254. Replace this values according to your own network settings and desires.

4) Configure Xen

The xend daemon employs xend-config.sxp to determines the parameters that Xen should use.

Personally I choose to disable dom0 ballooning, define the dom0 assignable memory and change the keyboard layout (I've changed mine to pt):

  1. # vim /etc/xen/xend-config.sxp (dom0-min-mem 2048) (enable-dom0-ballooning no) (keymap 'pt')

I've restricted the amount of memory assigned to dom0 to 2048 MB. In my case the dom0 is headless and all the hard work is to be done by the non-privileged virtual machines as such I've opted for a comfortable amount of memory to be assigned to the dom0, 2048 MB (2 GB). To this end GRUB needs to pass the appropriate command as the hypervisor boots:

  1. # echo 'GRUB_CMDLINE_XEN="dom0_mem=2G,max:2G"' >> /etc/default/grub
  2. # update-grub2


5) Prioritise Xen boot

By default Wheezy's GRUB lists and boots regular kernels and afterwards the Xen hypervisor.

Assuming that the computer is to be running Xen all the time it advisable to change this behaviour and increase Xen's GRUB boot priority so that it's the first on the list and boots by default.

The Debian way to do this is to used dpkg-divert like so:

  1. # dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen
  2. # update-grub2

To undo this necessary:
 
  1. # dpkg-divert --rename --remove /etc/grub.d/20_linux_xen
  2. # update-grub2


6) Create and install Windows 8 HVM domU

The xm toolstack uses configuration files that define the domain meaning that we need to create a configuration file for our guest VM:

  1. # vim /etc/xen/win8-x64.cfg kernel = 'hvmloader' builder = 'hvm' vcpus = '4' memory = '4096' disk = ['file:/srv/xen/domains/win8-x64.img,hda,w',
    'file:/srv/xen/images/Windows8-ReleasePreview-32bit-English.iso,hdc:cdrom,r'] name = 'win8-x64' vif = [ 'mac=00:16:3E:51:20:4C,bridge=br0,model=e1000' ] on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart' boot = 'dc' acpi = '1' apic = '1' viridian = '1' xen_platform_pci='1' sdl = '0' vnc = '1' vnclisten = '0.0.0.0' vncpasswd = '' stdvga = '0' usb = '1' usbdevice = 'tablet'

Do note that a MAC address must be assigned to the virtual interface. The 00:16:3e MAC block is reserved for Xen domains, do the last three digits may be randomly filled in (hex values 0-9 and a-f only).

In this HowTo I'm using file based storage which implies using the dd command to create what will be the domU hard drive. To create a 40GB .img file:

  1. # dd if=/dev/zero of=/srv/xen/domains/win8-x64.img bs=1M count=40960

If you're using LVM use 'phy:/dev/mapper/win8-x64,hda,w' (change according to your own target logical volume) instead of 'file:/srv/xen/domains/win8-x64.img,hda,w'.

For more on the options that the domain configuration file accepts refer to xmdomain.cfg.

There are 2 options when it comes to actually installing Windows 8 on the virtual machine. One method consists in using VNC to connect to the guest virtual machine and installing the operating system from whatever computer you have with a graphical desktop environment. In alternative, one can use VGA Passthrough for the install process altogether.

Choose one of the methods, though the VNC method is preferable as it eases troubleshooting and it's the one documented bellow. To use the VGA Passthrough method jump to step 7 of the HowTo and issue xm create win8-x64.cfg.

After defining Windows 8 domU configuration file execute it and connect through VNC to install Windows 8:

  1. # xm create win8-x64.cfg
  2. $ vncviewer 192.168.1.5

If running a GUI on dom0 simply vncviwer 127.0.0.1, however if running from a networked computer replace the localhost with the IP of the said networked computer (192.168.1.5 for example).

Proceed to do a Windows install, shutdown the guest VM and backup the .img for future use. To shutdown the Windows 8 HVM domU either use guest's shutdown button or issue:

  1. # xm destroy win8-x64

xm list can be used to find out the domain Id and use it as argument for xm destroy, for example:

  1. xm listName         ID Mem VCPUs State   Time(s)Domain-0     0 4096 8     r-----   34476.9win8-x64     4 4096 4     -b----   301.0
  2. xm destroy 4

Also comment out the cdrom line so that the virtual machine doesn't boot into the Windows installation cdrom every time it boots. For security reasons it is best to disable VNC.

  1. # vim /etc/xen/win8-x64.cfg kernel = 'hvmloader' builder = 'hvm' vcpus = '4' memory = '4096' disk = ['file:/srv/xen/domains/win8-x64.img,hda,w',
    #'file:/srv/xen/images/Windows8-ReleasePreview-32bit-English.iso,hdc:cdrom,r'] name = 'win8-x64' vif = [ 'mac=00:16:3E:51:20:4C,bridge=br0,model=e1000' ] on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart' boot = 'dc' acpi = '1' apic = '1' viridian = '1' xen_platform_pci='1' sdl = '0' vnc = '0' vnclisten = '0.0.0.0' vncpasswd = '' stdvga = '0' usb = '1' usbdevice = 'tablet'

7) Assign devices for PCI Passthrough

A domU can be made aware and directly access and use PCI devices with full privileges. To accomplish that the PCI devices need to be hidden from the dom0 and not be forwarded to any other domUs.

Using the xm toolstack this is achieved loading the pci_stub kernel module, identifying the PCI devices that are to be forwarded, unbinding the device from dom0 and bind it to pci_stub thus allowing it to be assigned in the domU config file.

  1. # lspci 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) 00:16.3 Serial controller: Intel Corporation 7 Series/C210 Series Chipset Family KT Controller (rev 04) 00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 04) 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.6 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 7 (rev c4) 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4) 00:1f.0 ISA bridge: Intel Corporation Q77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) 02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection 03:03.0 FireWire (IEEE 1394): LSI Corporation FW322/323 [TrueFire] 1394a Controller (rev 70)

I'll be forwarding 00:02.0 VGA compatible controller, 00:1b.0 Audio device and 00:1d.0 USB controller. To know the exact numbering of the devices run lspci -n:

  1. # lspci -n 00:00.0 0600: 8086:0150 (rev 09) 00:02.0 0300: 8086:0162 (rev 09) 00:14.0 0c03: 8086:1e31 (rev 04) 00:16.0 0780: 8086:1e3a (rev 04) 00:16.3 0700: 8086:1e3d (rev 04) 00:19.0 0200: 8086:1502 (rev 04) 00:1a.0 0c03: 8086:1e2d (rev 04) 00:1b.0 0403: 8086:1e20 (rev 04) 00:1c.0 0604: 8086:1e10 (rev c4) 00:1c.6 0604: 8086:1e1c (rev c4) 00:1d.0 0c03: 8086:1e26 (rev 04) 00:1e.0 0604: 8086:244e (rev a4) 00:1f.0 0601: 8086:1e47 (rev 04) 00:1f.2 0106: 8086:1e02 (rev 04) 00:1f.3 0c05: 8086:1e22 (rev 04) 02:00.0 0200: 8086:10d3 03:03.0 0c00: 11c1:5811 (rev 70)

For each PCI to be forwarded create a pci-stub Id, unbind it from the dom0 and bind to pci-stub. xm pci-list-assignable-devices is useful in confirming if the device has been added to the pool of devices that can be assigned to a guest domain.

  1. # modprobe pci_stub
  2. # echo "8086 1e26" > /sys/bus/pci/drivers/pci-stub/new_id
  3. # echo "0000:00:1d.0" > /sys/bus/pci/devices/0000\:00\:1d.0/driver/unbind
  4. # echo "0000:00:1d.0" > /sys/bus/pci/drivers/pci-stub/bind
  5. # xm pci-list-assignable-devices 0000:00:1d.0
  6. # echo "8086 0162" > /sys/bus/pci/drivers/pci-stub/new_id
  7. # echo "0000:00:02.0" > /sys/bus/pci/devices/0000\:00\:02.0/driver/unbind
  8. # echo "0000:00:02.0" > /sys/bus/pci/drivers/pci-stub/bind
  9. # xm pci-list-assignable-devices 0000:00:02.0 0000:00:1d.0
  10. # echo "8086 1e20" > /sys/bus/pci/drivers/pci-stub/new_id
  11. # echo "0000:00:1b.0" > /sys/bus/pci/devices/0000\:00\:1b.0/driver/unbind
  12. # echo "0000:00:1b.0" > /sys/bus/pci/drivers/pci-stub/bind
  13. # xm pci-list-assignable-devices 0000:00:02.0 0000:00:1b.0 0000:00:1d.0

Do note that the devices won't be available in the dom0, that's why typical VGA Passthrough setups involve 2 or more graphics cards forwarding the more powerful to the domU alongside an USB controller and audio (I'll cover Secondary Display Adapter PCI passthrough in a future post). In this case only the CPU's integrated GPU is present so as soon has 00:02.0 VGA compatible controller is hidden for the dom0 it can't be used by it and thus the only way to be access is via another computer using SSH for example.

Update the domU's configuration file with the devices that are to be used and start the Windows 8 domU by issuing xm create win8-x64.cfg.

  1. # vim /etc/xen/win8-x64.cfg kernel = 'hvmloader' builder = 'hvm' vcpus = '4' memory = '4096' disk = [ 'file:/srv/xen/domains/win8-x64.img,hda,w', #'file:/srv/xen/images/Windows8-ReleasePreview-32bit-English.iso,hdc:cdrom,r' ] name = 'win8-x64' vif = [ ',mac=00:16:3E:51:20:4C,bridge=br0,model=e1000' ] on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart' boot = 'dc' acpi = '1' apic = '1' viridian = '1' xen_platform_pci='1' sdl = '0' vnc = '0' vnclisten = '0.0.0.0' vncpasswd = '' stdvga = '0' usb = '1' usbdevice = 'tablet' pci = [ '00:1d.0', '00:1b.0' , '00:02.0' ]
  2. # xm create win8-x64.cfg

Also consider creating a script to automate loading pci-stub and unbinding/binding the  PCI devices.

8) Install GPLPV drivers

Developed by James Harper, GPLPV drivers allow swapping the QEMU emulated devices for paravirtualized devices. With these new devices I/O speeds are improved as Windows will use the network and block backend drivers present in the dom0.

Signed GPLPV drivers are available at http://wiki.univention.de/index.php?title=Installing-signed-GPLPV-drivers. Windows 8 32-bit can be downloaded at http://apt.univention.de/download/addons/gplpv-drivers/gplpv_Vista2008x32_signed_0.11.0.356.msi while 64-bit drivers are available at http://apt.univention.de/download/addons/gplpv-drivers/gplpv_Vista2008x64_signed_0.11.0.356.msi.
Vista2008 in the drivers' installer filename means that it applies to the recent Windows releases (e.g. Vista/2008/7/8).

9) Advices and impressions

I've written some pieces of advices and impressions over using XEN over at opensource-impressions.blogspot.com with some specific Intel HD4000 VGA Passthrough so make sure to read them.

Wheezy's stock kernel doesn't contain xen-acpi-processor as it was introduced during Linux kernel 3.4.x as such power management is lacking. In this scenario, consider compiling the kernel or in alternative installing Debian Experimental's kernel. The latter option implies editing /etc/apt/sources.list to cover the experimental branch and instructing apt-get to install the kernel package from the experimental distribution.

Xen is at version 4.1.3 in Debian Wheezy's repository however the recent 4.2.0 release is already available in the experimental branch. And with the 4.2.0 comes xl, the new toolstack set to replace xm.

In the coming weeks I'll update this HowTo with the required steps to combine Debian Experimental Xen and Linux kernel packages with the testing distribution.

Update 05/07/2013:
I've moved on from HD4000 VGA passthrough and am currently using PCI passthrough on 2 AMD cards (HD7850 and HD5450) with much success. The HD5450 is assigned to a Fedora 19 domU while the HD7850 is assigned to a Windows 8 domU (Catalyst drivers aren't installed) for my gaming fix. 
Stability of the setup: rock solid. dom0 is typically online for 1 to 2 months without rebooting, the Fedora 19 is hardly rebooted or shutdown while the Windows 8 domU is shutdown at night. Aside from these domU another Windows 8 domU is allways on.

37 comments:

Unknown said...

Very good to see this working! I have been trying to pass through my HD2000 iGPU (i7-2600) for about a year now with little success. I have an HD5850 that I pass through for gaming and CAD work which works very well, so I will have to try your method.

You should look into upgrading to xen 4.2.0 or at least look into using the xl toolstack rather than xm. In xl all you have to do to bind a device to pciback is:

sudo xl -f pci-assignable-add xx:xx.x

tangram said...

Hi,

Thanks for posting :)

Most issues with passthrough come dome to hardware namely motherboard and buggy BIOS implementation in regards to VT-d. For instance, in the current generation of Intel motherboards for Ivy Bridge CPUs, Q77 is the only chipset with official support for VT-d however there are several Z77 motherboards out there with advertised support; some ASROCK Z77 have positive all over the net but other such as Gigabyte are too buggy in regards to VT-d.

Regarding the toolstack... well I started with Xen 4.1 and xl at the time didn't achieved feature parity with xm (this still holds in some aspects even in 4.2). Aside from that I'm just lazy: stock Wheezy kernel and Xen just work with xm and pci_stub so I haven't bothered to try xl.

It's on my todo list though ;)

Regards,
tangram.

Juha said...

Very interesting article. How is the passthrough support for AMD IGPs? I am thinking of getting the new Trinity APU but have not been able to find out any information of the passthrough support with Xen. Is this an Intel-only feature?

tangram said...

Hi there,

Regarding AMD iGPU haven't read any reports. Discrete AMD cards work great though.

I do know that a few years back Intel submitted some iGPU passthrough patchs that made their way into Xen.

Regards,
tangram

Anonymous said...

Hi, Great writeup thanks. I also want to use PCI passthrough for my HD4000 video card (so I can install and use TMT5 Bluray Player). As you say, the Dom0 will not have a video card to use. So another poster recommends telnetting into the Dom0 in order to control it. I have a WinXP Domu installed and running. I only have WiFi access to the internet and no matter how hard I try I can't get the DomU to use it correctly. I was thinking about PCI passthrough for the WiFi card, but if I did that then I would not be able to Telnet into the DomU. Any instruction, help, or pointers on getting the Wifi to properly bridge to the DomU would be really appreciated. THANKS!!!

Archit Gupta said...

Please be patient as i have more than a few queries that i would like to have an answer of... before i do this virtualisation setup on my only laptop

I have a laptop with Core i5 3210m and AMD Radeon 7670m on it with only 4 GB RAM. The graphics card is a discrete one with AMD dynamic switchable graphics. Also in linux i can confirm that i see two graphics cards

What do you think, will it be possible to run Ubuntu (host) + Win8 (guest) on it. Will 4GB RAM suffice

Also after VGA passthrough will audio devices, USB devices work on both OSes simulraneously?

Will i be able to run say MATLAB(a scientific software) on Ubuntu(host) and browse internet on Win8 (guest) on my laptop and switch to any on the fly?

Thanks in advance..
Archit

tangram said...

Hi,

First you have to make sure that i5 3210m and your laptop's motherboard do in fact support VT-d. I suggest using Intel's ARK page or Google to find the spec sheet and make sure.

I'm not familiar with laptops with dynamic switchable graphics and whether these can be passed. You'd have to investigate for reports of similar attempts. Typically standalone AMD graphics cards are capable of being passed (strong emphasis on typically) but yours is the case on a laptop GPU.

PCI passthrough on a laptop is at the very least troublesome. As soon as you pass the device the host won't be able to access it and it isn't possible to share keyboard and mouse between dom0 and domU.

Bottom line, I doubt it is feasible.

Regards.

tangram said...

As for the anonymous poster with the WinXP setup...

Don't use Telnet. Please don't. SSH is so much better and was created to replace Telnet and similar insecure approaches.

I don't know how many Wifi cards you have...

If you have two you can use one for the dom0 and try to pass the other to the domU. If you only have one you'll have to investigate whether you can create a wireless bridge, have a look at Debian's network documentation on how to setup bridges.

Anonymous said...

Thanks for the post. I'm going to give this a shot now. For reference, I have a Gigabyte Z77N-Wifi with an Intel i7-3770S. Both support VT-d. I spent quite awhile attempting to get Intel 4000 graphics passthrough working with both ESXi 5.0 Update 2 and 5.1 Update 1 and a Windows 7 VM. No luck with either. The devices appear in the VM, but I was unable to successfully output video using them.

Seasky said...

Thank you for the article. I just following another to passthrough VGA but without luck.

I am going to setup a HTPC + LAMP server. I will use
* AMD APU
* ITX Motherboard (with 1 PCI Slot)
* PCI RAID CARD

In this setup, it have a on board display only.

I tried to create domU by using the config file, but it show the "parallel0 console" on boot screen.

I tried to create domU by using virt-manager, it could start successfully and install win xp without problem but fail to install ati driver.

The Xp find 2 VGA controller, 1 PCI
Device, 1 Audio Device on High Definition Audio Bus
DBut the catalyst driver cannot de

After I install windows, it detected 3 PCI , I install the catalyst drive. However, it should a small triangle and the error code is 10.

What should I check for it?

tangram said...

Hi,

First of all I', not sure that AMD APU support AMD-Vi "AMD I/O Virtualization Technology" or IOMMU (AMD equivalent to Intel's VT-d). I remember going through the specs for AMD's APU lineup in mid 2012 and at the time they didn't support AMD-Vi which is required for PCI / VGA passthrough.

Aside from that you motherboard also needs to support AMD-Vi.

Run dmesg | grep AMD-Vi and check the output.

Second it's not very clear from your post what you're trying to accomplish. VGA passthrough of primary (on-board) GPU? Or secondary PCI passthrough of a discrete GPU?

Also you don't mention distro, Xen version, domU configs, etc. And also you mention using virt-manager...

My suggestion: take baby steps. Check for proper HW support, then setup a domU with any passthrough (PCI or VGA), backup the working domU image/lvm and only then experiment with VGA or PCI passthrough.

Finally do take into account that PCI/VGA passthrough on Xen or other hypervisors for that matter is extremely experimental and hardware dependent.

Best of luck,
tangram.

tangram said...

Regarding Mr. Anonymous post on August 29... This is all about Xen. I don't have any experience with ESXi.

Btw Z77 chipset doesn't officially support VT-d. Double check hardware support.

Jo Mills said...

Many thanks for the article.

Do you happen to know whether moving to Wheezy backports so as to pick up xen-linux-system-3.12-0.bpo.1-amd64 cures the problems with xen-pciback and 4.1.4-3+deb7u1?

Best regards,

Jo.f

tangram said...

Hi Jo,

I'm running Wheezy backports for a couple months and I've experimented with crashes with Linux 3.11 and 3.12 regardless of xen version.

To have working PCI passthrough I'm running 3.10.11-1~bpo70+1 and 4.1.4-2_amd64 as anything above does work (namely 4.1.4-3+deb7u1_amd64 and 4.1.4-3_amd64).

Regards,
tangram.

Jo Mills said...

Hi tangram,

Many thanks for the very quick reply. May I just check for clarity, you said:

"as anything above does work (namely 4.1.4-3+deb7u1_amd64 and 4.1.4-3_amd64)"

Did you really mean "does work" or should this be "does not work"? I ask because I cannot get 4.1.4-3_amd64 to work with pciback and you seem to be quite explicit that you are using 4.1.4-2_amd64 with pciback.

Thanks again for your help,

Best regards,

Jo.

tangram said...

It should read doesn't work.

Jo Mills said...

Hi tangram,

Again, many many thanks for the very speedy reply. I will have a look through the Jessie packages and see if I can find whether this pciback bug has been fixed in them. If so, I'll probably give Jessie a try.

Thanks again for your help,

Best regards,

Jo.

Jo Mills said...

Hi tangram,

Apologies for posting again, but I have looked at upgrading to Jessie and decided it would be safer to follow the route you chose, "(namely 4.1.4-3+deb7u1_amd64 and 4.1.4-3_amd64)".

I believe the .debs I will need would be:
(1) xen-hypervisor-4.1-amd64_4.1.4-2_amd64.deb
(2) xenstore-utils_4.1.4-2_amd64.deb
(3) xen-system-amd64_4.1.4-2_amd64.deb
(4) xen-linux-system-3.10-3-amd64_3.10.11-1_amd64.deb

The first three of these I have found from various FTP sites. The fourth one, whilst seemingly very common, doesn't seem to exist anymore.

Please would you let me know if I am looking for the correct files, and if so, do you know where I might find my fourth file.

Sorry to trouble you again,

Regards,

Jo.

Jo Mills said...

Hi tangram,

This is driving me nuts ;-) I have (I think) tracked down all the relevant packages and resolved all necessary conflicts. My system shows:

ii xen-hypervisor-4.1-amd64 4.1.4-2 amd64 Xen Hypervisor on AMD64
ii xen-system-amd64 4.1.4-2 amd64 Xen System on AMD64 (meta-package)
ii xenstore-utils 4.1.4-2 amd64 Xenstore utilities for Xen
ii xen-linux-system-3.10-0.bpo.3-amd64 3.10.11-1~bpo70+1 amd64

"xm dmesg" shows >> (XEN) I/O virtualisation enabled



"lspci -k" shows >>

03:00.0 Ethernet controller: VIA Technologies, Inc. VT6105/VT6106S [Rhine-III] (rev 86)
Subsystem: D-Link System Inc DFE-530TX rev C
Kernel driver in use: via-rhine
03:01.0 Ethernet controller: VIA Technologies, Inc. VT6105/VT6106S [Rhine-III] (rev 86)
Subsystem: D-Link System Inc DFE-530TX rev C
Kernel driver in use: via-rhine
03:02.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)
Subsystem: Intel Corporation Device 34d0
Kernel driver in use: e1000


/etc/modules is >>

loop
xen_pciback


/etc/modprobe.d/xen-pciback.conf is >>

install via-rhine /sbin/modprobe pciback ; /sbin/modprobe --first-time --ignore-install via-rhine
options xen-pciback hide=(0000:03:01.0)


lsmod | egrep "pciback|via" shows >>

xen_pciback 37748 0
via_rhine 23052 0
mii 12675 1 via_rhine


and yet when I try

~# xm pci-list-assignable-devices

it shows nothing at all.

Time for a beer I think.

Any suggestions, corrections to what I've done, etc. all very gratefully received. I couldn't have got this far without your help.

Best regards,

Jo.

tangram said...

Hi,

I'm not at my computer so I can't check the details but AFAIR simply install Xen as you would normally and then use dpkg to install the old 4.1.4-2.

Regards.

tangram said...

I'm off for the weekend but I'll post further details when possible.

tangram said...

Hi Jo,

Right now this is what I have installed in terms of packages:
ii libxen-4.1 4.1.4-3+deb7u1 amd64 Public libs for Xen
ii libxenstore3.0 4.1.4-3+deb7u1 amd64 Xenstore communications library for Xen
ii xen-hypervisor-4.1-amd64 4.1.4-2 amd64 Xen Hypervisor on AMD64
ii xen-linux-system-3.10-0.bpo.3-amd64 3.10.11-1~bpo70+1 amd64 Xen system with Linux 3.10 on 64-bit PCs (meta-package)
ii xen-system-amd64 4.1.4-3+deb7u1 amd64 Xen System on AMD64 (meta-package)
ii xen-utils-4.1 4.1.4-3+deb7u1 amd64 XEN administrative tools
ii xen-utils-common 4.1.4-3+deb7u1 all Xen administrative tools - common files
ii xenstore-utils 4.1.4-3+deb7u1 amd64 Xenstore utilities for Xen

Aside from that xen_pciback is loaded in /etc/modules. I'm using options xen-pciback passthrough=1 in /etc/modprobe.d/xen-pciback.conf though AFAIR is worked perfecty with vpci also.

However I don't use xen-pciback as I just couldn't make it work properly (this is due to the fact that xen-pciback is compiled as a module in Debian kernels). Instead I'm using pci_stub on a rudimentary script:

#!/bin/sh

# Load pci_stub kernel module
modprobe pci_stub

# Unload drivers for the specified PCI devices
echo "8086 10d3" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:03:00.0" > /sys/bus/pci/devices/0000\:03\:00.0/driver/unbind
echo "0000:03:00.0" > /sys/bus/pci/drivers/pci-stub/bind
echo "1002 6819" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:01:00.0" > /sys/bus/pci/devices/0000\:01\:00.0/driver/unbind
echo "0000:01:00.0" > /sys/bus/pci/drivers/pci-stub/bind
echo "1002 aab0" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:01:00.1" > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind
echo "0000:01:00.1" > /sys/bus/pci/drivers/pci-stub/bind
echo "8086 1e2d" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:00:1a.0" > /sys/bus/pci/devices/0000\:00\:1a.0/driver/unbind
echo "0000:00:1a.0" > /sys/bus/pci/drivers/pci-stub/bind

# List PCI devices that are available to be assigned to domains
xm pci-list-assignable-devices

# Wait for 10 seconds then start domU
sleep 10
xm create /etc/xen/windows-8.1.cfg

Finally fglrx, radeon and emu10k1 are blacklisted.

Jo Mills said...

Hi tangram,

Again, many thanks for the very quick reply. I just got back in from work and I've tried your suggestion (also I looked at http://wiki.xen.org/wiki/VTd_HowTo). All my package versions now match with your list.


Q: When you issue the command

# xm pci-list-assignable-devices

Do you get a list of devices? I get nothing. I can see from "lspci -k" that my card is using pci-stub

03:01.0 0200: 1106:3106 (rev 86)
Subsystem: 1186:1403
Kernel driver in use: pci-stub

Probably I've just done something silly, I think I'll leave it for now and look again tomorrow.

Many thanks again for you quick reply.

Best regards,

Jo.

tangram said...

Yep, it lists all the devices that can be assigned.

Are you sure your hardware is VT-D capable?

J said...

Hi tangram,

Many thanks for the quick reply, my "xm dmesg" gives:

(XEN) Domain heap initialised
(XEN) Processor #0 7:7 APIC version 20
(XEN) Processor #2 7:7 APIC version 20
(XEN) Processor #1 7:7 APIC version 20
(XEN) Processor #3 7:7 APIC version 20
(XEN) IOAPIC[0]: apic_id 5, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2660.081 MHz processor.
(XEN) Initing memory sharing.
(XEN) Intel VT-d supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation not enabled.
(XEN) Intel VT-d Interrupt Remapping not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) HVM: ASIDs disabled.
(XEN) HVM: VMX enabled
(XEN) Brought up 4 CPUs


Maybe this has been my major problem all along? I previously checked in "xm dmesg" that it said "I/O virtualisation enabled", which it clearly does, but the messages above that don't look too hopeful! I've got to go to work now, but I'll check through my BIOS settings again tonight.

Thanks again for the quick reply,

Best regards,

Jo.


Jo Mills said...

Hi tangram,

I've just been through the BIOS settings and I cannot see anything
that looks as though it ought to be set that isn't set.

Motherboard : Intel S3200SHV
Processor : Intel Core 2 (Quad)
BIOS : S3200X38.86B.00.000052

BIOS Processor settings:

Enhanced Intel (R) SpeedStep Tech Enabled
Core Multi-Processing Enabled
Intel (R) Virtual Technology Enabled
Inter (R) VT for Directed I/O Enabled
Execute Disable Bit Enabled
Multi-Thread Support in MPS Table Enabled


Powered off server, removed power cord, waited 60 seconds, powered
server back up.


modprobe pci_stub
echo "1106 3106" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:03:01.0" > /sys/bus/pci/devices/0000:03:01.0/driver/unbind
echo "0000:03:01.0" > /sys/bus/pci/drivers/pci-stub/bind

xm pci-list-assignable-devices

==> nothing at all.

The Ethernet card is:

03:01.0 0200: 1106:3106 (rev 86)
Subsystem: 1186:1403
Kernel driver in use: pci-stub


03:01.0 Ethernet controller: VIA Technologies, Inc. VT6105/VT6106S [Rhine-III] (rev 86)
Subsystem: D-Link System Inc DFE-530TX rev C
Kernel driver in use: pci-stub


It looks to me like it ought to work, the kernel log says:

Feb 18 19:57:42 green kernel: [ 23.584056] xenbr0: port 1(eth0) entered forwarding state
Feb 18 19:57:45 green kernel: [ 25.767712] Event-channel device installed.
Feb 18 20:00:51 green kernel: [ 212.718641] pci-stub 0000:03:01.0: claimed by stub


Which looks hopeful, but still "no cigar". Oh hum, ...

tangram said...

Hmm... you're using the xm toolstack and xend right?

Jo Mills said...

Hi tangram,

I believe that I am.

# ps ax | grep xend
3858 ? S 0:00 /usr/bin/python /usr/lib/xen-4.3/bin/xend start
3859 ? SLl 0:05 /usr/bin/python /usr/lib/xen-4.3/bin/xend start


xl is available, but the xm toolstack is currently in use.


Yesterday I upgraded my system to Debian Jessie to see if that improved matters, it didn't.

I'll have a read through
http://wiki.xen.org/wiki/XL
tonight and see if moving to xl makes the required difference.

Many thanks for your help,

Best regards,

Jo.

Jo Mills said...

Hi tangram,

** BRILLIANT **

You are truly a marvel, so many many thanks for this. I have upgraded to the xl toolstack and now I have:

/etc/modules
------------
xen_pciback passthrough=1

depmod -a
re-boot

Then to try from the command line:
# BDF=0000:03:01.0
# [ ! -e /sys/bus/pci/devices/$BDF/driver/unbind ] || echo -n $BDF > /sys/bus/pci/devices/$BDF/driver/unbind
# echo -n $BDF > /sys/bus/pci/drivers/pciback/new_slot
green:~# echo -n $BDF > /sys/bus/pci/drivers/pciback/bind
# xl pci-assignable-list
0000:03:01.0

It has appeared (and using pciback).

Many many thanks for all your help and patience. I owe you a huge favour, so if there is something I can do for you in return, let me know.

Best regards,

Jo.




tangram said...

Nice :)

Did you upgrade to Jessie? Did you upgrade xen?

ozieboy said...

Hi Tangram,
Thanks for the Awesome guide. Please see if you can help resolve my problem.
I had to use debian testing netinstall in order to have my network adapter ( intel haswell NUC) detected and do the i-get install and did not nstallation.So with testing (Jessie), I just installed Xen using aptget any of the other packages like4.1.4-2_amd64 or anything.
I started using Xm and then Xl and eventally get the IGP Intel HD 5000 show up in windows 8.1 Vm. I installed intel device drivers and and it shows up as intel hd 5000 in device manager.
BUT, it do not start and gives resources allocation error ( error 12) no resources available to be assigned to the device.
So it seems that I am almost there but yet so far away.
Can you please think if anything or the right order to go about the whole install.
Should I get an old image of debian 7.1 stable wheezy and try with that?
Please advise, if you can.
Thanks and Regards,
Ozieboy.

Jo Mills said...

Hi tangram,

I did upgrade to Jessie. I did a dist-upgrade last Thursday (just before I tried your tip about moving to the xl toolstack) so I am also running the Jessie xen packages.

Thanks again for your help,

Best regards,

Jo.

tangram said...

Ozieboy,

I'm still running wheezy so I'm not aware of the state of Xen VGA passthrough on Jessie. From what Jo Mills reported (comments prior to yours) it is working with the xl toolstack and default Xen packages for Jessie.

You've mentioned error 12 but didn't provide details on wether that's on Windows or Xen.

For the moment I don't plan on upgrading to Jessie so I won't be able to provide much help there.

Regards,
tangram.

Ozieboy. said...

Hi Tangram,

Thanks for your quick reply, really appreciate that.
The error is in the windows 8.1 Vms device manager.
(cannot start intel hd 50000,because there is no resources available t be assigned to it )
I am happy to use Wheezy, my question is if i need to find an old wheezy 7.1 to get the same kernal and xen versions as yours or the latest wheezy 7.4 would be fine too.
Also did you do a Wheezy netinst or use the standard ISO to install without network?Because If I use wheezy 7.4 netinst,it do not pick up my intel network adapter and so not able to do the install.
Best Regards,
Ozieby

tangram said...

Hi,

You should use the latest Wheezy installer.

I'm currently using a full updated Wheezy with Wheezy backports for kernel.

The only thing particular is that I'm using still xen-hypervisor-4.1-amd64 4.1.4-2 instead of latest package. You can get that from Debian's archives.

Regards,
tangram.

Matthew Durkin said...

Hi Tangram,
I'm trying to do something very similar but have come a bit unstuck - I wonder if you might have got this far and found a solution!...
I'm running on an asrock z87e-itx board which supports vt-d (enabled in bios). I have a 4770 CPU which also supports vt-d. And I'm hoping to pass the IGP graphics (Intel 4600) through to a windows 8 guest.
My dom0 is ubuntu 14.04 - mostly because I've used ubuntu server for years so was an obvious choice, but I could very easily switch to Debian if it's likely to make a difference.
Thus far I have windows 8 installed, and can pass through the VGA device as follows:

echo "0000:00:02.0" > /sys/bus/pci/devices/0000:00:02.0/driver/unbind
echo "0000:00:02.0" > /sys/bus/pci/drivers/pciback/new_slot
echo "0000:00:02.0" > /sys/bus/pci/drivers/pciback/bind
xl pci-assignable-list
0000:00:02.0

When I then boot windows 8 I see a new device, can install the drivers, and can then see the Intel HD Grahpics 4600 in device manager with a yellow exclamation and a code 12 error (This device cannot find enough free resources that it can use).
I also see 'Other Devices' - PCI device with no drivers installed (code 28) which I assume is related in some way.

on the dom0 console I also notice when I run
xl dmesg
I get
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.

I wonder if DMA passthrough is my issue. If yes, how do I enable?

Anyway, as you seem to have managed to get onboard IGP passed through, albeit on a different system, I'm very keen to know if you might have come across this before and if yes, if you have any pointers!!

Thanks,
Matt

tangram said...

Hi Matthew,

You're seriously deviating from this HowTo (written for Debian Wheezy, using Xen 4.1.3, the xm toolstack and pci-stub). When I ran this setup I never ran into any problems with Windows' device manager and performance was quite good.

I must confess that I haven't followed VGA passthrough since late 2012 as I move into PCI passthrough and kept on using Xen 4.1.3 + xm toolstack.

I suggest consulting Xen's Wiki on VGA passthrough with xl toolstack.

Best of luck,
tangram