TrueNAS Build Nvidia vGPU Driver extensions (systemd-sysext)

Running TrueNAS on Proxmox VE is the great way to get great VM and storage management at same time, especially for homelab users. Because TrueNAS’s built-in VM management is difficult to use, offers limited VM options, and doesn’t allow for system setting modifications. While it uses Incus as its VM backend after 25.04, the TrueNAS WebUI and middleware still need a lot of work.

And vGPU has become a prevalent method for enabling VMs on Proxmox VE to utilize a GPU card for video encoding and decoding. However, TrueNAS, when run as a VM guest, defaults to using standard Nvidia drivers, not the specialized Grid drivers (NVIDIA’s commercial drivers). Consequently, while some Tesla cards might correctly load nvidia-smi, they often can’t actually use their encoding and decoding units with these default drivers.

The good news is that starting with TrueNAS SCALE 24.10+, TrueNAS now uses systemd-sysext to load Nvidia drivers that are pre-packaged during compilation. This extension can load when needed, Which also means you can overwrite and install any custom driver without using install-dev-tools, avoiding system modifications. Additionally, recovery is quick after TrueNAS updates.

How It Works#

The core principle behind the packaged extension source code is quite straightforward:

It compares the contents of self.chroot and self.chroot_base directories.
It extracts any new or modified files.
Only files under usr/ (excluding usr/src/) are kept; others are deleted.
It cleans up self.chroot by removing files and empty directories that don’t belong to the extension.
An extension-release.d file is added to identify the extension.
Finally, self.chroot is packaged into a SquashFS file and saved to dst_path.

Here’s the Python code:

1
    def build_extension(self, name, dst_path):
2
        changed_files = [
3
            os.path.relpath(filename, self.chroot)
4
            for filename in map(
5
                lambda filename: os.path.join(os.getcwd(), filename),
6
                run(
7
                    ["rsync", "-avn", "--out-format=%f", f"{self.chroot}/", f"{self.chroot_base}/"],
8
                    log=False,
9
                ).stdout.split("\n")
10
            )
11
            if os.path.abspath(filename).startswith(os.path.abspath(self.chroot))
12
        ]
13

14
        sysext_files = [f for f in changed_files if f.startswith("usr/") and not (f.startswith("usr/src/"))]
15

16
        for root, dirs, files in os.walk(self.chroot, topdown=False):
17
            for f in files:
18
                path = os.path.relpath(os.path.abspath(os.path.join(root, f)), self.chroot)
19
                if path not in sysext_files:
20
                    os.unlink(os.path.join(root, f))
21

22
            for d in dirs:
23
                try:
24
                    os.rmdir(os.path.join(root, d))
25
                except NotADirectoryError:
26
                    os.unlink(os.path.join(root, d))  # It's a symlink
27
                except OSError as e:
28
                    if e.errno == errno.ENOTEMPTY:
29
                        pass
30
                    else:
31
                        raise
32

33
        os.makedirs(f"{self.chroot}/usr/lib/extension-release.d", exist_ok=True)
34
        with open(f"{self.chroot}/usr/lib/extension-release.d/extension-release.{name}", "w") as f:
35
            f.write("ID=_any\n")
36

37
        run(["mksquashfs", self.chroot, dst_path, "-comp", "xz"])

Specifically for the Nvidia driver, this is the relevant part:

1
    def download_nvidia_driver(self):
2
        prefix = "https://us.download.nvidia.com/XFree86/Linux-x86_64"
3

4
        version = get_manifest()["extensions"]["nvidia"]["current"]
5
        filename = f"NVIDIA-Linux-x86_64-{version}-no-compat32.run"
6
        result = f"{self.chroot}/{filename}"
7

8
        self.run(["wget", "-c", "-O", f"/{filename}", f"{prefix}/{version}/{filename}"])
9

10
        os.chmod(result, 0o755)
11
        return result
12

13
    def install_nvidia_driver(self, kernel_version):
14
        driver = self.download_nvidia_driver()
15

16
        self.run([f"/{os.path.basename(driver)}", "--skip-module-load", "--silent", f"--kernel-name={kernel_version}",
17
                     "--allow-installation-with-running-driver", "--no-rebuild-initramfs"])
18

19
        os.unlink(driver)

So, modifying this is actually quite simple:

Replace the download link.
Adjust the execution options (Grid drivers have different options).

PS: When choosing drivers, pay attention to kernel compatibility. For instance, with the current 25.04 version (kernel version 6.12.15), if you’re using a 16.x driver (the last usable driver for Tesla P4), your grid driver version needs to be greater than 16.9.

Here’s how the modified code would look:

1
    def download_nvidia_driver(self):
2
        # i don't where i can find the grid driver
3
        prefix = "<put your grid download url without filename in here>"
4
        #useless
5
        #version = get_manifest()["extensions"]["nvidia"]["current"]
6
        filename = f"<filename of your grid driver>"
7
        result = f"{self.chroot}/{filename}"
8

9
        self.run(["wget", "-c", "-O", f"/{filename}", f"{prefix}/{filename}"])
10

11
        os.chmod(result, 0o755)
12
        return result
13

14
    def install_nvidia_driver(self, kernel_version):
15
        driver = self.download_nvidia_driver()
16
        # fix option
17
        self.run([f"/{os.path.basename(driver)}", "--silent", f"--kernel-name={kernel_version}"])
18

19
        os.unlink(driver)

Building the Extension#

In here, I’ll use 25.04.0 as an example.

The official documentation already provides some hints, but here are some key things to note:

Install Dependencies#

(Nothing need to notice)

1
sudo apt install build-essential debootstrap git python3-pip python3-venv squashfs-tools unzip libjson-perl rsync libarchive-tools

Select the Tags Branch#

Use the tag branch instead of release/xxxxx

1
# tag name is TS-<version>
2
git clone -b TS-25.04.0 https://github.com/truenas/scale-build.git

Configure Environment Variables#

The code reads environment variables to determine the compiled “version” and “Train”. So, if you don’t specify them, you’ll only be able to compile the Master train.

1
export TRUENAS_TRAIN="TrueNAS-SCALE-Fangtooth"
2
export TRUENAS_VERSION="25.04.0"

Start the Build#

1
make checkout
2

3
make packages
4

5
make update

Once the build is complete, you’ll need to mount the compiled rootfs.squashfs and extract its contents.

1
mkdir -p ./tmpfile/rootfs
2
mount ./tmp/update/rootfs.squashfs ./tmpfile/rootfs

At this point, your nvidia.raw extension should be available:

1
ls -al ./tmpfile/rootfs/usr/share/truenas/sysext-extensions/nvidia.raw

Overwriting the Existing Driver#

You’ll need to replace the nvidia.raw file on your running TrueNAS system at /usr/share/truenas/sysext-extensions/nvidia.raw with the one you just compiled.

First, you need to make the /usr dataset writable:

1
zfs set readonly=off boot-pool/ROOT/<system version>/usr
2
# For 25.04.0, it would be:
3
zfs set readonly=off boot-pool/ROOT/25.04.0/usr

PS: If you’ve previously enabled the Nvidia driver via the WebUI, you might need to run systemd-sysext unmerge first. This is because systemd-sysext will set /usr as read-only.

Overwrite it!

1
cp /the-path-you-upload/nvidia.raw /usr/share/truenas/sysext-extensions/nvidia.raw

After you’ve copied the file, simply run:

1
systemd-sysext merge
2
# Don't forget to restart docker service
3
systemctl restart docker

And, It should work.

1
nvidia-smi