The Tesla P40 is a seductive piece of hardware: 24GB of VRAM for a fraction of the cost of a modern RTX card. After three weeks of fighting with it, I realized that the “budget” part of the equation doesn’t include the cost of my sanity. I spent more time debugging QEMU assertion errors and PCI address shifts than I did actually running models.

If you’re looking to put a P40 in a Proxmox node to run LLMs, you’re likely trying to fit larger models like Qwen2.5:32B into VRAM without spending four figures on an A100 or a 3090. It’s a viable path, but the standard way of doing things (GPU passthrough to a VM) is a recipe for instability with this specific card.

What I Tried First: The “Clean” Architecture

My first instinct was to follow the standard Proxmox pattern: isolate the GPU using vfio-pci and pass it through to a dedicated Ubuntu VM. I’ve done this before, and usually, it’s the right move for isolation. I had my IOMMU groups sorted and the hostpci line configured in the VM config. I wanted a clean separation where the host remained a pure hypervisor and the VM handled the NVIDIA drivers and the Ollama runtime.

It worked for about four hours. Then the P40 decided it didn’t want to exist anymore.

The Tesla P40 lacks Function Level Reset (FLR). In a virtualized environment, this means that if the VM crashes, the driver hangs, or you simply reboot the guest, the GPU doesn’t actually reset to a clean state. The next time you try to boot the VM, you get a QEMU assertion error or a “Device is already in use” message.

I found myself hard-rebooting the entire physical node just to get the GPU to respond again. I’ve written about GPU passthrough gotchas before, but the P40 is particularly aggressive about breaking the happy path.

PCI address instability also became a factor. After a few reboots and some BIOS tweaks, the card shifted addresses, and my VM config became a lie. I was essentially playing a game of whack-a-mole with my hardware topology, as detailed in my post on GPU PCI address instability.

The Solution: Host-Level Inference

I stopped trying to be “architecturally clean” and decided to run the GPU directly on the Proxmox host. Running production-ish workloads on the hypervisor is usually a sin, but the P40 is too unstable in a VM to justify the overhead. By running on the host, I eliminate the QEMU translation layer and the FLR reset issues. If the driver crashes, I can reload the kernel module without power-cycling the entire server.

Here is exactly how I moved from a broken passthrough setup to a stable host-level inference engine.

1. Cleaning the Slate

First, I stripped the GPU out of the VM and killed the VFIO isolation. If you’ve already pinned your GPU to vfio-pci, you need to undo that. This is a critical step because if vfio-pci is still holding the device, the NVIDIA driver will fail to initialize the card, and you’ll see the “Unable to determine the device handle” error in the logs.

# Remove the PCI device from the VM config
qm set <VM_ID> --hostpci0 ''

# Blacklist vfio to stop it from grabbing the card at boot
echo "blacklist vfio_pci" | sudo tee /etc/modprobe.d/vfio.conf
echo "blacklist vfio" | sudo tee -a /etc/modprobe.d/vfio.conf

# Update initramfs to ensure the blacklist is applied at boot
update-initramfs -u
reboot

2. Host Driver Installation

I installed the NVIDIA 535 drivers directly on the Proxmox host. I chose 535 because it’s stable with the P40’s Pascal architecture. Newer drivers sometimes introduce regressions for these older datacenter cards, and 535 is the current “sweet spot” for stability.

sudo apt update
sudo apt install nvidia-driver-535

# Verify the card is seen and the driver is loaded
sudo nvidia-smi

When I ran nvidia-smi for the first time, I saw the expected 24GB of VRAM, but the card was in “Compute” mode. For some users, you might need to ensure the card isn’t stuck in a weird state. If the card doesn’t show up, check dmesg | grep -i nvidia. If you see “RM Init Adapter failed,” it’s usually a sign of a power issue or a poorly seated riser.

3. Deploying Ollama as a Systemd Service

Instead of wrapping Ollama in a container on the host (which adds another layer of driver mapping pain and potential nvidia-container-toolkit version mismatches), I deployed it as a systemd service. This ensures it starts on boot and has direct access to the GPU without runtime overhead.

I created a service file at /etc/systemd/system/ollama.service:

[Unit]
Description=Ollama
After=network.target

[Service]
User=ollama
Group=ollama
WorkingDirectory=/opt/ollama
ExecStart=/opt/ollama/ollama serve
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_KEEP_ALIVE=30s"
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

I set OLLAMA_HOST=0.0.0.0 so my other nodes in the cluster could hit the API, and OLLAMA_KEEP_ALIVE=30s to ensure the model unloads from VRAM quickly when not in use. This is important because if you’re experimenting with different models, you don’t want a 15GB model sitting in VRAM preventing you from loading a different one.

Troubleshooting the “Ghost GPU”

Even on the host, the P40 can be temperamental. The most common error I hit was the driver version mismatch. This happens when the NVIDIA kernel module is updated via apt but the userspace libraries aren’t synced. You’ll run nvidia-smi and get a message saying the driver version is mismatched with the library.

The fix is to reload the modules, but since the GPU is in use, you often have to stop the Ollama service first.

sudo systemctl stop ollama
sudo rmmod nvidia_uvm
sudo rmmod nvidia_modeset
sudo rmmod nvidia
sudo modprobe nvidia
sudo systemctl start ollama

If the modules are locked and won’t unload, you’re back to a reboot. This is why I pinned the driver version in my package manager to prevent automatic updates from breaking the runtime in the middle of the night.

VRAM Management and the KV Cache Trap

Having 24GB of VRAM is great, but it isn’t infinite. When running a model like Qwen2.5:32B (quantized), the model weights take up a large portion of the memory. The remaining space is used for the KV cache, which stores the context of the conversation.

I noticed that as the context window grew, inference speed would plummet, and eventually, the process would crash with an out-of-memory (OOM) error. The P40 uses older GDDR5 memory, which is significantly slower than the HBM2 found in newer Tesla cards or the GDDR6X in 3090s.

To manage this, I had to be specific about the context window. If I set the context to 32k tokens, I would lose roughly 4GB of VRAM just for the cache. I found that capping the context at 8k tokens for most tasks kept the system stable and the response times acceptable. If you need larger contexts, you’ll need to look into privacy-routed LLM inference patterns to offload some of the load or use more aggressive quantization.

Closing the Monitoring Gap

The biggest problem with running a GPU on the host is that Proxmox doesn’t natively show GPU metrics in the web UI. You can see CPU and RAM, but the GPU is a black box. I didn’t want to SSH into the node and run nvidia-smi every ten minutes to check temperatures.

Since the P40 is a passive card, it relies entirely on external airflow. If your fan curve is wrong, the card will thermal throttle at 80C and your tokens per second will drop by 50%. I solved this by deploying nvidia_gpu_exporter as a DaemonSet in my Kubernetes cluster (since I have a hybrid setup where the GPU node also acts as a K8s worker).

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-gpu-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: nvidia-gpu-exporter
  template:
    metadata:
      labels:
        app: nvidia-gpu-exporter
    spec:
      containers:
      - name: exporter
        image: nvidia/gpu-exporter:latest
        ports:
        - containerPort: 9835
        resources:
          limits:
            nvidia.com/gpu: 1
      tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "gpu"
        effect: "NoSchedule"

This exporter feeds metrics into Prometheus and Grafana. I set up an alert to ping me if the GPU temperature exceeds 75C. This is the only way to ensure the card doesn’t cook itself in a chassis that wasn’t designed for server-grade passive GPUs.

Why This Actually Works

The shift from VM to host works because it removes the abstraction layers that the P40 struggles with. The P40 was designed for a bare-metal Linux environment in a data center, not for a QEMU virtual machine. By running on the host, we avoid the “reset” problem entirely. When the host boots, the card is initialized once. When the Ollama service restarts, it just re-initializes the CUDA context without needing to reset the hardware.

There is a tradeoff here: security. You are now running a third-party binary (Ollama) as a system service on your hypervisor. In a corporate environment, this is a non-starter. In a homelab, the tradeoff is acceptable because the alternative is a system that crashes every few hours.

If you are building similar AI infrastructure for a business, I’d suggest looking at infrastructure consulting to figure out a more isolated but stable approach, perhaps using a dedicated bare-metal node for inference rather than a hypervisor.

Lessons Learned and Caveats

I learned that the documentation for “budget GPUs” usually ignores the cooling and power requirements. The P40 is a power-hungry beast that produces an immense amount of heat. If you don’t have a high-static-pressure fan blowing directly into the heatsink, you will hit thermal limits in minutes. I had to 3D print a shroud to force air through the card.

Another surprise was the driver versioning. I spent two days thinking my card was dying when it was actually just a mismatch between the kernel module and the NVIDIA libraries. Always check dmesg before assuming hardware failure.

If I did this again, I would start with the host-level installation. I wasted a week trying to force the “correct” architectural pattern of VM isolation on hardware that simply doesn’t support it gracefully. The P40 is a great way to get 24GB of VRAM on a budget, but only if you’re willing to treat the hypervisor as a workstation.

Watch out for the nvidia-container-toolkit if you decide to move Ollama into a Docker container on the host. You’ll need to ensure the runtime is set to nvidia in your daemon.json, otherwise, the container will start but won’t see the GPU, leading to the same CPU-inference slowness I saw in my early Ollama on Kubernetes experiments.

The Tesla P40 remains a viable tool for local LLM work, provided you accept that it’s a piece of legacy hardware that requires a bit of manual care to keep running.