TrueNAS Build Nvidia vGPU Driver extensions (systemd-sysext)

A lot of folks are opting to run TrueNAS on Proxmox VE these days. Why? Well, TrueNAS’s built-in VM management can be a bit clunky – even with Incus, the underlying tech isn’t quite seamless, and you often hit snags with CLI commands acting up in the Instance UI. To get around this and supercharge things like RDP desktops or video encoding, many are cleverly using vGPU passthrough to share a single GPU across several virtual machines.

However, TrueNAS by default uses the standard Nvidia drivers, not the specialized Grid drivers (NVIDIA’s commercial drivers). While some Tesla cards might correctly load nvidia-smi, they often can’t actually use their encoding and decoding units with the default drivers.

The good news is that starting with TrueNAS SCALE 24.10+, TrueNAS now uses systemd-sysext to load Nvidia drivers that are pre-packaged during compilation. These extensions are only loaded when needed.

How It Works#

The core principle behind the packaged extension source code is quite straightforward:

It compares the contents of self.chroot and self.chroot_base directories.
It extracts any new or modified files.
Only files under usr/ (excluding usr/src/) are kept; others are deleted.
It cleans up self.chroot by removing files and empty directories that don’t belong to the extension.
An extension-release.d file is added to identify the extension.
Finally, self.chroot is packaged into a SquashFS file and saved to dst_path.

Here’s the Python code:

1
    def build_extension(self, name, dst_path):
2
        changed_files = [
3
            os.path.relpath(filename, self.chroot)
4
            for filename in map(
5
                lambda filename: os.path.join(os.getcwd(), filename),
6
                run(
7
                    ["rsync", "-avn", "--out-format=%f", f"{self.chroot}/", f"{self.chroot_base}/"],
8
                    log=False,
9
                ).stdout.split("\n")
10
            )
11
            if os.path.abspath(filename).startswith(os.path.abspath(self.chroot))
12
        ]
13

14
        sysext_files = [f for f in changed_files if f.startswith("usr/") and not (f.startswith("usr/src/"))]
15

16
        for root, dirs, files in os.walk(self.chroot, topdown=False):
17
            for f in files:
18
                path = os.path.relpath(os.path.abspath(os.path.join(root, f)), self.chroot)
19
                if path not in sysext_files:
20
                    os.unlink(os.path.join(root, f))
21

22
            for d in dirs:
23
                try:
24
                    os.rmdir(os.path.join(root, d))
25
                except NotADirectoryError:
26
                    os.unlink(os.path.join(root, d))  # It's a symlink
27
                except OSError as e:
28
                    if e.errno == errno.ENOTEMPTY:
29
                        pass
30
                    else:
31
                        raise
32

33
        os.makedirs(f"{self.chroot}/usr/lib/extension-release.d", exist_ok=True)
34
        with open(f"{self.chroot}/usr/lib/extension-release.d/extension-release.{name}", "w") as f:
35
            f.write("ID=_any\n")
36

37
        run(["mksquashfs", self.chroot, dst_path, "-comp", "xz"])

Specifically for the Nvidia driver, this is the relevant part:

1
    def download_nvidia_driver(self):
2
        prefix = "https://us.download.nvidia.com/XFree86/Linux-x86_64"
3

4
        version = get_manifest()["extensions"]["nvidia"]["current"]
5
        filename = f"NVIDIA-Linux-x86_64-{version}-no-compat32.run"
6
        result = f"{self.chroot}/{filename}"
7

8
        self.run(["wget", "-c", "-O", f"/{filename}", f"{prefix}/{version}/{filename}"])
9

10
        os.chmod(result, 0o755)
11
        return result
12

13
    def install_nvidia_driver(self, kernel_version):
14
        driver = self.download_nvidia_driver()
15

16
        self.run([f"/{os.path.basename(driver)}", "--skip-module-load", "--silent", f"--kernel-name={kernel_version}",
17
                     "--allow-installation-with-running-driver", "--no-rebuild-initramfs"])
18

19
        os.unlink(driver)

So, modifying this is actually quite simple:

Replace the download link.
Adjust the execution options (Grid drivers have different options).

PS: When choosing drivers, pay attention to kernel compatibility. For instance, with the current 25.04 version (kernel version 6.12.15), if you’re using a 16.x driver (the last usable driver for Tesla P4), your grid driver version needs to be greater than 16.9.

Here’s how the modified code would look:

1
    def download_nvidia_driver(self):
2
        # i don't where i can find the grid driver
3
        prefix = "<put your grid download url without filename in here>"
4
        #useless
5
        #version = get_manifest()["extensions"]["nvidia"]["current"]
6
        filename = f"<filename of your grid driver>"
7
        result = f"{self.chroot}/{filename}"
8

9
        self.run(["wget", "-c", "-O", f"/{filename}", f"{prefix}/{filename}"])
10

11
        os.chmod(result, 0o755)
12
        return result
13

14
    def install_nvidia_driver(self, kernel_version):
15
        driver = self.download_nvidia_driver()
16
        # fix option
17
        self.run([f"/{os.path.basename(driver)}", "--silent", f"--kernel-name={kernel_version}"])
18

19
        os.unlink(driver)

Building the Extension#

In here, I’ll use 25.04.0 as an example.

The official documentation already provides some hints, but here are some key things to note:

Install Dependencies#

(Nothing need to notice)

1
sudo apt install build-essential debootstrap git python3-pip python3-venv squashfs-tools unzip libjson-perl rsync libarchive-tools

Select the Tags Branch#

Use the tag branch instead of release/xxxxx

1
# tag name is TS-<version>
2
git clone -b TS-25.04.0 https://github.com/truenas/scale-build.git

Configure Environment Variables#

The code reads environment variables to determine the compiled “version” and “Train”. So, if you don’t specify them, you’ll only be able to compile the Master train.

1
export TRUENAS_TRAIN="TrueNAS-SCALE-Fangtooth"
2
export TRUENAS_VERSION="25.04.0"

Start the Build#

1
make checkout
2

3
make packages
4

5
make update

Once the build is complete, you’ll need to mount the compiled rootfs.squashfs to extract its contents. Here’s how:

1
mkdir -p ./tmpfile/rootfs
2
mount ./tmp/update/rootfs.squashfs ./tmpfile/rootfs

At this point, your nvidia.raw extension should be available:

1
ls -al ./tmpfile/rootfs/usr/share/truenas/sysext-extensions/nvidia.raw

Overwriting the Existing Driver#

You’ll need to replace the nvidia.raw file on your running TrueNAS system at /usr/share/truenas/sysext-extensions/nvidia.raw with the one you just compiled.

First, you need to make the /usr dataset writable:

1
zfs set readonly=off boot-pool/ROOT/<system version>/usr
2
# For 25.04.0, it would be:
3
zfs set readonly=off boot-pool/ROOT/25.04.0/usr

PS: If you’ve previously enabled the Nvidia driver via the WebUI, you might need to run systemd-sysext unmerge first. This is because systemd-sysext otherwise controls /usr as read-only.

Overwrite it!

1
cp /the-path-you-upload/nvidia.raw /usr/share/truenas/sysext-extensions/nvidia.raw

After you’ve copied the file, simply run:

1
systemd-sysext merge
2
# Don't forget to restart docker
3
systemctl restart docker

And, It should work.

1
nvidia-smi