I need to pass a Google Coral mPCIe device for TensorFlow processing to my Ubuntu 20.04+ VM with a docker container running Frigate
The motherboard I am using is Intel based and does not have an m.2 port but does have an mPCIe slot which was used for the WiFi/BT device and since they are not used in my virtual setup I swapped them out with the mPCIe Google Coral Adapter
Note: A USB device is available but has a high price tag especially when ordering 4 units
So, Here is how I enabled and passed it to my ubuntu VM
Enable IOMMU in Proxmox
Open your bootloader kernel command line config file
For grub & Intel CPU’s:
nano /etc/default/grub
Replace the “GRUB_CMDLINE_LINUX_DEFAULT” with
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
Reboot
Verify IOMMU is enabled
Run
dmesg | grep -e DMAR -e IOMMU
There should be a line that looks like “DMAR: IOMMU enabled”. If there is no output, something is wrong
Check for PCI device being detected
root@proxmox:~# lspci -nnk | grep 089a
02:00.0 System peripheral [0880]: Device [1ac1:089a]Subsystem: Device [1ac1:089a]
The 02.00.0 is the important part. This will change based upon your system and/or PCIe port used
Make sure 02:00.0 is in it’s own iommu group. If you dont get anything returned, you may need to go back and check enabling IOMMU
root@proxmox:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/34/devices/0000:7f:0a.1 /sys/kernel/iommu_groups/34/devices/0000:7f:0a.2 /sys/kernel/iommu_groups/6/devices/0000:00:04.0 /sys/kernel/iommu_groups/62/devices/0000:80:04.2 /sys/kernel/iommu_groups/90/devices/0000:ff:10.1 /sys/kernel/iommu_groups/24/devices/0000:02:00.0
This will show a long list but you can grep for the device bus ID if you know it and then grep again the iommu group – commands below 😉
root@proxmox:~# find /sys/kernel/iommu_groups/ -type l | grep 04:
/sys/kernel/iommu_groups/24/devices/0000:02:00.0
root@proxmox:~# find /sys/kernel/iommu_groups/ -type l | grep /24/
/sys/kernel/iommu_groups/24/devices/0000:02:00.0
Blacklist the Device
Create a new file at /etc/modprobe.d/blacklist-apex.conf and add these lines:
root@proxmox:~# nano /etc/modprobe.d/blacklist-apex.conf
blacklist gasket blacklist apex options vfio-pci ids=1ac1:089a
Save the file and then then run the following to apply the changes
root@proxmox:~# update-initramfs -u -k all
The command above will take a while to run. Once completed restart ProxMox
Verify that the apex and gasket modules did not load by running this:
root@proxmox:~# lsmod | grep apex
It should not return anything
Add PCIe Device to VM
Once rebooted and confirmed all in order as above
Stop the VM, add the PCI device to the VM (where 100 is the VM ID in Proxmox)
root@proxmox:~# qm set 100 -hostpci0 02:00.0
thingsIforget 