PCI passthrough of Nvidia geforce

This is something, I have done some years ago while playing with cryptominers. The basic idea is to give a virtual machine access the the physical hardware. I have seen it done with network interface cards also.

Get the PCI connection values

 lspci | grep NVIDIA
 01:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] (rev a1)
 01:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)

Ensure that you are not using the device.

At least the nouveau kernel module really did not like to loose a device, and crashed all graphics.

 root@ioto:~# lsmod | grep nouveau
 nouveau              2412544  2
 mxm_wmi                16384  1 nouveau
 i2c_algo_bit           16384  2 i915,nouveau
 drm_ttm_helper         16384  1 nouveau
 ttm                    86016  3 drm_ttm_helper,i915,nouveau
 drm_kms_helper        307200  2 i915,nouveau
 drm                   634880  20 drm_kms_helper,drm_ttm_helper,i915,ttm,nouveau
 wmi                    36864  3 intel_wmi_thunderbolt,mxm_wmi,nouveau
 video                  57344  2 i915,nouveau
 button                 24576  1 nouveau
 root@ioto:~# rmmod nouveau
 rmmod: ERROR: Module nouveau is in use

If you are, you need to blacklist and rebuild initrd

See eg.g this guide. The easy solution is to blacklist the nouveau driver and rebooting.

as root
```
 echo "blacklist nouveau" > /etc/modprobe.d/blacklist-nouveau.conf
 update-initramfs -u
```

Set up vfio-pci.

This related to isolation of IO, memory, interrupts and such. For more info see here, here and here.

Get ids for vfio-pci:

 lspci -n | grep 01:00
 root@ioto:~# lspci -n | grep 01:00
 01:00.0 0300: 10de:1c02 (rev a1)
 01:00.1 0403: 10de:10f1 (rev a1)

Set module parameters

 # cat /etc/modprobe.d/nvidia_vfio.conf
 vfio vfio_iommu_type1 vfio_pci ids=10de:1c02,10de:10f1

Enable load of vfio-pci at boot time

 # cat /etc/modules-load.d/vfio-pci.conf 
 vfio-pci

Reboot for changes to take effect

And when it is done booting, verify that the driver is not installed.

Verify that nouveau is gone

 lsmod | grep nouveau

verify that the nvidia card i using vfio-pci

 # lspci -k | grep -i nvidia -A4
 01:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] (rev a1)
     Subsystem: ASUSTeK Computer Inc. GP106 [GeForce GTX 1060 3GB]
     Kernel driver in use: vfio-pci
     Kernel modules: nouveau
 01:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)
     Subsystem: ASUSTeK Computer Inc. GP106 High Definition Audio Controller
     Kernel driver in use: vfio-pci
     Kernel modules: snd_hda_intel

Set up vagrant

 # cat Vagrantfile 
 Vagrant.configure("2") do |config|
     config.vm.box = "debian/bullseye64"

     config.vm.provider :libvirt do |libvirt|
         libvirt.pci :domain => '0x0000', :bus => '0x01', :slot => '0x00', :function => '0x0'
         libvirt.pci :domain => '0x0000', :bus => '0x01', :slot => '0x00', :function => '0x1'
     end
 end