Proxmox PCIe passthrough

I need to pass a Google Coral mPCIe device for TensorFlow processing to my Ubuntu 20.04+ VM with a docker container running Frigate

The motherboard I am using is Intel based and does not have an m.2 port but does have an mPCIe slot which was used for the WiFi/BT device and since they are not used in my virtual setup I swapped them out with the mPCIe Google Coral Adapter

Note: A USB device is available but has a high price tag especially when ordering 4 units

So, Here is how I enabled and passed it to my ubuntu VM

Enable IOMMU in Proxmox

Open your bootloader kernel command line config file

For grub & Intel CPU’s:

nano /etc/default/grub

Replace the “GRUB_CMDLINE_LINUX_DEFAULT” with

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

Reboot

Verify IOMMU is enabled

Run

dmesg | grep -e DMAR -e IOMMU

There should be a line that looks like “DMAR: IOMMU enabled”. If there is no output, something is wrong

Check for PCI device being detected

root@proxmox:~# lspci -nnk | grep 089a
02:00.0 System peripheral [0880]: Device [1ac1:089a]Subsystem: Device [1ac1:089a]

The 02.00.0 is the important part. This will change based upon your system and/or PCIe port used

Make sure 02:00.0 is in itโ€™s own iommu group. If you dont get anything returned, you may need to go back and check enabling IOMMU

root@proxmox:~# find /sys/kernel/iommu_groups/ -type l

/sys/kernel/iommu_groups/34/devices/0000:7f:0a.1 /sys/kernel/iommu_groups/34/devices/0000:7f:0a.2 /sys/kernel/iommu_groups/6/devices/0000:00:04.0 /sys/kernel/iommu_groups/62/devices/0000:80:04.2 /sys/kernel/iommu_groups/90/devices/0000:ff:10.1 /sys/kernel/iommu_groups/24/devices/0000:02:00.0

This will show a long list but you can grep for the device bus ID if you know it and then grep again the iommu group – commands below ๐Ÿ˜‰

root@proxmox:~# find /sys/kernel/iommu_groups/ -type l | grep 04:

/sys/kernel/iommu_groups/24/devices/0000:02:00.0

root@proxmox:~# find /sys/kernel/iommu_groups/ -type l | grep /24/

/sys/kernel/iommu_groups/24/devices/0000:02:00.0

Blacklist the Device

Create a new file at /etc/modprobe.d/blacklist-apex.conf and add these lines:

root@proxmox:~# nano /etc/modprobe.d/blacklist-apex.conf
blacklist gasket
blacklist apex
options vfio-pci ids=1ac1:089a

Save the file and then then run the following to apply the changes

root@proxmox:~# update-initramfs -u -k all

The command above will take a while to run. Once completed restart ProxMox

Verify that the apex and gasket modules did not load by running this:

root@proxmox:~# lsmod | grep apex

It should not return anything

Add PCIe Device to VM

Once rebooted and confirmed all in order as above

Stop the VM, add the PCI device to the VM (where 100 is the VM ID in Proxmox)

root@proxmox:~# qm set 100 -hostpci0 02:00.0