Sign in Subscribe
Proxmox

Why I'm Ripping NVIDIA Out of My Proxmox Host

Leon SCOTT

• 8 min read
Share
Why I'm Ripping NVIDIA Out of My Proxmox Host
Proxmox 9-1

I have spent more evenings than I'd like to admit keeping an NVIDIA GPU alive on a Proxmox host. Not doing anything clever with it. Just keeping the driver building across kernel updates.

It works, I have LXC passthrough and can play with AI rather well.

This week I finally accepted that the problem isn't my setup, isn't a bug I can wait out, and isn't something a better script can fix. It's structural. So the RTX 3070 is coming out, and an AMD card is going in.

If you run NVIDIA on a Proxmox host and share it into LXC containers, you have probably already met this wall, or you're about to. Here's the whole story, with the evidence, so you can decide before you sink another weekend into it.

The setup

My host pxe runs Proxmox VE 9.1 (Debian 13 / Trixie underneath). One RTX 3070 lives on the host. The NVIDIA kernel modules stay on the host only; unprivileged LXC containers get the GPU by binding the /dev/nvidia* device nodes. Jellyfin transcodes through it, and I've run the usual local-AI bits and pieces against it (Ollama, a bit of CUDA tinkering).

The driver is the Debian-packaged one from trixie-backports: nvidia-kernel-dkms + nvidia-driver, currently 550.163.01. That was a deliberate choice. I want the driver apt-tracked and DKMS-managed, not a .run blob I have to babysit by hand. Package management is the whole point of running Debian underneath.

That deliberate choice is exactly where the trap is hidden.

The NVIDIA-on-Proxmox treadmill

NVIDIA's kernel module is out-of-tree. It does not ship with the Linux kernel; it has to be compiled against whatever kernel you're running, every time that kernel changes. DKMS automates the rebuild, and when it works you barely notice. When it doesn't, a routine apt run can leave you with a GPU that nvidia-smi no longer sees.

This is not a me-problem. It's a documented, recurring class of pain. The general failure mode -- a kernel update silently breaking the NVIDIA install so the card stops being detected -- is written up as one of the most common Proxmox GPU-passthrough pitfalls (Mr.PlanB, Medium). Search the Proxmox forum and you'll find the same shape of thread going back years: drivers not surviving a kernel bump, DKMS not rebuilding, headers missing for the new kernel.

For a long time the fix was just discipline: keep pve-headers installed so DKMS has something to build against, run full-upgrade not upgrade, verify dkms status before you reboot. Annoying, but livable.

Then the kernel moved somewhere the Debian-packaged 550 driver simply cannot follow.

The specific wall: 550 can't build past ~6.18

Here's the part that turns "annoying" into "dead end."

The Debian-packaged NVIDIA driver in trixie-backports is 550.163.01. Per the Debian wiki, as of 2026-03-26 that driver no longer compiles on kernels v6.19 released to backports. It tops out somewhere around 6.18. It has no hope of building against kernel 7.0.

And it's not theoretical. A Proxmox user with a Quadro P2000 hit the exact DKMS build failure on 6.17.4-1-pve: nvidia-current/550.163.01 dying in nvidia-drm on a DRM API change (a drm_helper_mode_fill_fb_struct signature mismatch), ending in the classic bad exit status: 2 (Proxmox forum). Their question at the end is the same one I ended up asking myself: is the recommended approach just to stick on an old kernel if you need NVIDIA on the host?

That is why my host is pinned to the 6.17 kernel series. Not as a workaround I'm being lazy about -- as the correct state, because there is no newer Debian-packaged driver that builds higher. I checked every suite: trixie-backports, forky (testing), and sid (unstable) all still ship 550.163.01. Only experimental has anything newer (555), which is not something you put on a host you care about. There is currently no official, package-managed NVIDIA driver that builds on kernel 7.0.

Proxmox 9.2 forces the question

You can hold a kernel forever, right up until the distribution decides to move the default out from under you.

That's what 9.2 does. It ships the 7.0 kernel (Ubuntu 26.04 "Resolute" base) as the default. On my host, a security update to the 6.17 series quietly dragged in the entire 7.0 series through the proxmox-default-kernel meta-package -- and the NVIDIA DKMS autoinstall for 7.0 failed mid-apt, wedged dpkg with half-configured packages, and left a bootable-but-broken 7.0 kernel that GRUB wanted to boot by default.

Again: not just me. The live "NVIDIA drivers failing after upgrade to Proxmox 9.2.2 (kernel 7.0.2-6-pve)" thread shows the same dependency cascade, and a homelab user trying to go 6.17 -> 7.0 hitting a kernel panic, with the NVIDIA module (580.105.08 in their case) failing to build on 7.0.6-2-pve -- bad exit status: 2, same wall (Proxmox forum). There's a fresh twist on 7.0 too: Nova, the new Rust-based in-kernel driver meant to succeed Nouveau for GSP-firmware GPUs, can auto-enable on the upgrade and fight the NVIDIA module for the card.

The guard I built (and why it's a tourniquet, not a cure)

Because the silent meta-package pull nearly bricked my boot kernel, I wrote a guard into my host GPU script. Three independent layers:

  1. Preflight gate -- simulate apt full-upgrade, inspect every kernel/header package it would install, and block if any belongs to a kernel series I haven't explicitly approved.
  2. apt holds -- hold the series-selector meta-packages plus any unapproved-series kernel packages, surgically, so 6.17 security point-updates still flow but the 7.0 jump does not.
  3. Boot pin -- pin the running, known-good kernel via proxmox-boot-tool so even a bad install can't become the default boot target.

It works. It did its job. But be honest about what it is: it's a tourniquet that keeps me on an aging kernel indefinitely. Every layer exists to prevent forward motion because forward motion breaks the GPU. That's not a strategy, it's a holding pattern. The moment 6.17 ages out of the Proxmox repos, the holding pattern ends and I'm forced onto 7.0 anyway.

The realisation: it was never 580 vs 590

I'll cop to a wrong turn here, because it's instructive. I'd half-convinced myself the fix was a driver-branch question -- something like "580 breaks on 7.0, 590 builds, so migrate branches." That framing is a trap. It's about the upstream .run installer / CUDA-repo / Arch packaging world. It has nothing to do with what's actually on my host.

nvidia-smi on pxe says 550.163.01. I was never on 580 or 590. The real axis isn't the branch number, it's the packaging channel. I use the Debian repo package, and Debian's packaged driver is frozen at 550 across every sane suite, and 550 can't build past ~6.18. Full stop.

So the honest options, if I insist on staying NVIDIA:

  • Stay package-managed, stay pinned. Correct, but a dead end on a timer.
  • Wait for Debian to ship a newer packaged driver. Nothing in backports/forky/sid suggests it's close.
  • Leave official Debian packaging. Either the .run installer (the exact hand-managed blob I refuse to babysit) or an unofficial-but-still-apt/DKMS repo. There's a genuinely interesting one from Debian Developer Samuel Henrique -- an "nvidia-ai" repo on Debusine (Debian's PPA-equivalent) that packages current NVIDIA-open as proper .debs with DKMS, versioned to sort below official so you migrate back cleanly when Debian catches up. It's a real option. But it's one maintainer, single-machine-tested, and it would live on my hypervisor. That's a lot of trust for the one box everything else depends on.

Every NVIDIA path either freezes me in place or asks me to hand-manage an out-of-tree module forever. That's the actual product I've been buying with the green team on Linux: a permanent maintenance tax.

The fix that ends the argument: in-tree AMD

Here's the thing that reframes all of it. The entire guard/pin/DKMS saga exists because NVIDIA's module is out-of-tree. AMD's amdgpu driver is in the mainline kernel. It ships with the kernel it's built for. Upgrade to 7.0, 7.1, whatever -- the driver comes along automatically, already built, already matched. There is no DKMS step to fail, no headers to chase, no series to pin, no meta-package to ambush you.

The whole class of problem doesn't get solved. It ceases to exist.

And for my LXC-passthrough model it's actually simpler than NVIDIA: no host driver install beyond Mesa userspace; containers just bind /dev/dri/renderD128 for VAAPI (and /dev/kfd if you want ROCm compute). The notorious AMD "reset bug" people warn about is a VFIO/VM-passthrough problem -- it doesn't touch LXC, where everything shares the host kernel and you're binding device nodes, not resetting a card handed to a guest.

The honest AMD caveats

I'm not going to pretend it's free, because the forums are clear it isn't:

  • You can still walk back into DKMS hell if you ask for it. There's an amdgpu-dkms package -- AMD's out-of-tree Pro/ROCm stack via amdgpu-install -- and it breaks on kernel updates exactly like NVIDIA does. A Proxmox user watched amdgpu-dkms fail to build across kernels with the familiar bad exit status (Proxmox forum). The lesson: use the in-tree driver, do not install amdgpu-dkms unless you specifically need the Pro compute stack. If you install the out-of-tree stack, you've re-created the problem you left NVIDIA to escape.
  • ROCm on consumer Radeon can be fiddly. You may need HSA_OVERRIDE_GFX_VERSION for some tooling. The fallback is clean, though: llama.cpp's Vulkan backend runs on any modern Radeon and sidesteps ROCm entirely, so local inference isn't hostage to the ROCm support matrix.
  • AMD's video encoder used to be the weak link. Historically sub-par H.264, which matters because most Jellyfin clients pull H.264. This is the one genuinely important buying constraint, and it drives the card choice below.

The card

For a host GPU whose job is transcode plus light local AI -- not gaming -- the sweet spot is the Radeon RX 9060 XT 16GB (RDNA4). In Australia right now that's roughly AUD $500-$600 for the 16GB models (the 8GB variants are ~$450; ignore them, 8GB is what you're escaping from).

Why this specific card:

  • Transcode. AMD's encoder generations rank VCN5 (RX 9000 / RDNA4) clearly ahead of VCN4 (RX 7000), ahead of the older RX 6000/5000 blocks (Jellyfin AMD hardware docs). RDNA4 is where AMD's historically weak H.264 encoder finally gets good, and you get AV1 encode/decode on top. Buying a 7000-series to save a little would mean buying the older encoder.
  • AI headroom. 16GB is double the 3070's 8GB. For Ollama models that VRAM is the real upgrade, more than raw compute.
  • The whole point. In-tree amdgpu, so Proxmox 9.2 and its 7.0 kernel just work, and every future kernel bump is a non-event.

The RX 9070 / 9070 XT (roughly $800 / $970+) are gaming-flagship tier. A hypervisor doing transcode and occasional inference will never use that raster muscle, and the 9060 XT draws less (~150-180W vs the 3070's 220W), which is friendlier for a box that's on 24/7.

Bottom line

If you run NVIDIA on a Proxmox host via the Debian-packaged driver, understand what you actually own: a driver frozen at 550, a kernel you can't move past ~6.18, and a distribution that's moving its default to 7.0 out from under you. The scripts and guards I built are good engineering aimed at a problem that shouldn't exist. The out-of-tree module is the problem.

An in-tree AMD card doesn't out-compute the 3070 on paper. It does something better for a homelab host: it makes the maintenance tax disappear. I'll take "boring and it just keeps working" over "powerful and I'm pinned to an EOL kernel" every single time.

RX 9060 XT 16GB. Pull the green card. Reclaim the weekends.

Very frustrating to say the least. Hope this helps

#enoughsaid

Sources referenced: Debian NvidiaGraphicsDrivers wiki; Proxmox VE forum threads on the 550/6.17 DKMS failure, the 9.2.2 / kernel 7.0 NovaCore conflict, the amdgpu-dkms kernel-pin issue, and drivers not surviving kernel updates; Jellyfin AMD hardware-acceleration docs; Mr.PlanB's Proxmox passthrough pitfalls writeup. Verify current package versions and pricing before you buy – both move.