NVIDIA vGPU drivers on Flatcar Linux with RKE2

12 minute read

I’ve been slowly moving more of my homelab onto Kubernetes, and one of the more awkward pieces has been GPU support.

The target was simple enough: I wanted my Flatcar Linux RKE2 cluster to run workloads that need NVIDIA vGPU. In the happy-path version of this story, I’d install the NVIDIA GPU Operator, point it at the correct vGPU driver, and let it manage the driver daemonset, toolkit, device plugin, GPU feature discovery, and all the other moving parts.

That is still the goal, and it is mostly where I ended up. The bit in the middle was more interesting than expected.

Flatcar is a deliberately minimal container host. That’s part of why I like it for Kubernetes nodes, but it also means the usual “just install the driver on the host” approach doesn’t really fly. The NVIDIA GPU Operator wants to run a driver container. For normal distributions, NVIDIA publishes suitable driver container images. For my particular combination of Flatcar, RKE2, and NVIDIA vGPU, I needed images that matched the Flatcar version and kernel I was actually running.

So the problem became:

  1. get a Flatcar build environment into an OCI image;
  2. use that environment to build NVIDIA vGPU driver containers;
  3. produce tags in the format the GPU Operator already expects;
  4. keep those images current as Flatcar rolls forward.

The result is two public projects:

  • docker-flatcar-developer, which publishes Flatcar developer container images to GHCR as ghcr.io/andrewreid/flatcar-developer.
  • vgpu-driver-builder, which contains the vgpu-driver-operator, a controller that builds vGPU driver containers for Flatcar Linux and stores them in a registry of your choosing, ready for NVIDIA’s GPU Operator to consume.

Starting with the GPU Operator

I didn’t want to hand-roll daemonsets for driver loading, the container toolkit, GPU feature discovery, or device plugin management. The operator already knows how to do all of that. I also didn’t really love the Flatcar “proper” way of doing things, which seemed to be sysext-related and didn’t quite cover my particular use case involving an ageing vGPU-capable card, a Tesla P40.

The catch is that the GPU Operator constructs driver image tags based on the configured driver version and the node OS. On Flatcar, the runtime driver tag looks like this:

535.261.03-flatcar4593.2.0

For precompiled drivers, the kernel is included as well:

535.261.03-6.12.81-flatcar-flatcar4593.2.0

I could’ve built the images manually after every Flatcar upgrade, pushed them to my registry, and then hoped I remembered all the details next time. But that’s a pain, and frankly it misses an opportunity to unnecessarily overengineer something that barely has an excuse to exist in my homelab in the first place. Flatcar updates regularly, and driver images are cluster infrastructure. They should be declared and reconciled like everything else.

First missing piece: a Flatcar developer image

Flatcar provides a developer container image upstream, but not in the form I wanted to use from a normal Docker/BuildKit workflow. The developer environment is distributed as flatcar_developer_container.bin.bz2. Inside that is the root filesystem with the toolchain needed to build software against Flatcar.

I found what appears to be an abandoned project on GitHub that looked like it might do the trick, forked it, fiddled with it, rewrote most of it to use a more current build pipeline, and ended up with docker-flatcar-developer: a GitHub Workflow that turns Flatcar’s developer container into OCI images, published as ghcr.io/andrewreid/flatcar-developer.

For every Flatcar version it sees, it publishes two variants:

ghcr.io/andrewreid/flatcar-developer:<version>
ghcr.io/andrewreid/flatcar-developer:<version>-sources

The plain tag is the Flatcar developer root filesystem. The -sources tag adds the matching kernel sources, coreos-overlay, portage-stable, and a prepared /usr/src/linux tree suitable for building out-of-tree kernel modules.

The base image is deliberately boring:

# syntax=docker/dockerfile:1
FROM scratch
ADD rootfs.tar /

The slightly less boring work happens in the publishing workflow. It discovers the current Flatcar version for each channel, downloads the developer container, loop-mounts the partition, extracts rootfs.tar, and builds the OCI image. The sources image then layers on the matching kernel source:

ARG BASE_REGISTRY=ghcr.io
ARG BASE_IMAGE
ARG FLATCAR_VERSION
FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${FLATCAR_VERSION}

ARG FLATCAR_VERSION
ARG FLATCAR_TRACK

RUN emerge-gitclone \
    && export OVERLAY_VERSION="${FLATCAR_TRACK}-${FLATCAR_VERSION}" \
    && export PORTAGE_VERSION="${FLATCAR_TRACK}-${FLATCAR_VERSION}" \
    && git -C /var/lib/portage/coreos-overlay checkout "tags/$OVERLAY_VERSION" \
    && git -C /var/lib/portage/portage-stable checkout "tags/$PORTAGE_VERSION"

RUN emerge -gKq --jobs 4 --load-average 4 coreos-sources \
    || emerge -q --jobs 4 --load-average 4 coreos-sources

RUN cp /usr/lib64/modules/$(ls /usr/lib64/modules)/build/.config /usr/src/linux/ \
    && make -C /usr/src/linux modules_prepare \
    && cp /usr/lib64/modules/$(ls /usr/lib64/modules)/build/Module.symvers /usr/src/linux/

That gives me a reusable base for anything that needs to compile against a particular Flatcar release. NVIDIA vGPU drivers are just the first consumer.

Second missing piece: building driver images in-cluster

Once the Flatcar build environment existed, the next question was where to run the NVIDIA driver builds.

I wanted the build process to be:

  • declarative;
  • repeatable;
  • tied to the Flatcar versions present in the cluster;
  • able to push into a local/private registry;
  • decoupled from the GPU Operator itself.

That became the vgpu-driver-operator: 99% AI-constructed slop that seems to actually work and automates the process of building precompiled vGPU drivers for Flatcar releases as they’re published.

The operator watches a VGPUDriverImage custom resource. Given a list of NVIDIA driver versions, a source location for the .run files, and a target registry, it works out which images should exist and starts Kubernetes Jobs for the missing ones.

A typical resource looks like this:

apiVersion: vgpu.flatcar.io/v1alpha1
kind: VGPUDriverImage
metadata:
  name: vgpu-drivers
  namespace: vgpu-driver-operator
spec:
  driverVersions:
    - "535.261.03"

  source:
    type: s3
    uriTemplate: s3://vgpu-drivers/NVIDIA-Linux-x86_64-${DRIVER_VERSION}-grid.run
    credentialsSecretRef:
      name: s3-driver-storage-secret

  registry:
    repository: registry.example.com/vgpu-driver
    repositoryPrecompiled: registry.example.com/vgpu-driver-precompiled
    cacheRepository: registry.example.com/vgpu-driver-cache
    authSecretRef:
      name: private-registry-secret

  flatcar:
    discoverFromNodes: true
    trackChannels:
      - stable

  precompile: true

  retention:
    enabled: true
    keepPreviousFlatcarVersions: 2
    minAgeBeforeDelete: "168h"

The operator combines the requested driver versions with Flatcar versions from a few sources:

  • Flatcar nodes currently in the cluster;
  • explicit versions in the CRD;
  • tracked Flatcar release channels such as stable;
  • existing registry tags, for retention and garbage collection decisions.

For each required combination it creates a BuildKit Job. The job fetches the NVIDIA .run file from S3, builds the image, and pushes it to the configured registry.

Runtime images are tagged like this:

registry.example.com/vgpu-driver:535.261.03-flatcar4593.2.0

Precompiled images include the kernel:

registry.example.com/vgpu-driver-precompiled:535.261.03-6.12.81-flatcar-flatcar4593.2.0

Runtime builds vs precompiled builds

I ended up supporting two modes.

The runtime image is the simpler version. It contains the NVIDIA installer, the Flatcar build environment, and an entrypoint script. When the GPU Operator runs the driver pod on a node, the container compiles the NVIDIA kernel modules against that node’s kernel and loads them.

The Dockerfile is basically:

ARG FLATCAR_IMAGE_REPO=ghcr.io/andrewreid/flatcar-developer
ARG FLATCAR_IMAGE_SUFFIX=-sources
ARG FLATCAR_VERSION
ARG FLATCAR_IMAGE_REF=${FLATCAR_IMAGE_REPO}:${FLATCAR_VERSION}${FLATCAR_IMAGE_SUFFIX}
FROM ${FLATCAR_IMAGE_REF}

ARG DRIVER_VERSION
ENV DRIVER_VERSION=${DRIVER_VERSION} \
    NVIDIA_VISIBLE_DEVICES=void \
    DRIVER_TYPE=vgpu

RUN mkdir -p /drivers
COPY NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run /drivers/
COPY nvidia-driver /usr/local/bin/nvidia-driver
RUN chmod +x /usr/local/bin/nvidia-driver

ENTRYPOINT ["nvidia-driver"]

This works, and it’s flexible, but the driver pod has real work to do each time it lands on a node. That means slower rollouts and more moving parts at exactly the point where I’d rather the node be boring.

The precompiled image moves the kernel module build earlier. The operator builds an image for a specific Flatcar version, discovers the kernel version from the matching flatcar-developer:<version>-sources image, compiles the NVIDIA modules during the image build, and publishes the final image with the kernel in the tag.

The awkward bit is that the operator doesn’t necessarily know the kernel version in advance. Flatcar’s release metadata has changed over time, and relying on an external feed for the kernel string turned out to be brittle. The source of truth is the base image itself.

So the precompiled Dockerfile has a small discovery stage:

FROM ${FLATCAR_IMAGE_REF} AS kernel-discover

RUN set -eu \
    && set -- /lib/modules/* \
    && if [ "$#" -ne 1 ] || [ ! -d "$1" ]; then \
         echo "ERROR: expected exactly one kernel under /lib/modules" >&2; \
         exit 1; \
       fi \
    && printf '%s\n' "${1##*/}" > /kernel_version

FROM scratch AS kernel-discover-export
COPY --from=kernel-discover /kernel_version /kernel_version

The BuildKit job runs that target first and exports /kernel_version locally. It then runs the real build with KERNEL_VERSION passed as a build argument and uses that same value in the output tag.

That two-phase build solved a surprisingly annoying problem: the GPU Operator requires the kernel in the precompiled image tag, but the most reliable place to get the kernel is from inside the build environment.

The driver entrypoint

The driver container entrypoint has two jobs:

  • in build-precompiled mode, compile and stage the modules during the image build;
  • in normal init mode, prepare the mounted host root, install or sync modules, load the NVIDIA kernel modules, and start the vGPU userspace pieces.

The important part is that both paths share the same module build logic. If the image has precompiled modules for the running kernel, it uses them. If it doesn’t, a runtime image can still build them on the node.

In simplified form:

if precompiled_modules_available; then
    depmod -b "${NVIDIA_KMODS_DIR}" "${KERNEL_VERSION}"
    validate_precompiled_modules
    sync_modules_to_host
else
    prepare_kernel_env
    build_and_install_modules
    sync_modules_to_host
fi

modprobe -d "${NVIDIA_KMODS_DIR}" -a nvidia nvidia-uvm nvidia-modeset
nvidia-persistenced --persistence-mode
nvidia-gridd

The real script is more defensive than that. It validates module metadata with modinfo, checks vermagic against the kernel version, stages modules into the mounted host filesystem, and starts the vGPU services expected by NVIDIA’s stack.

One lesson from this work: for kernel modules, failing early is much nicer than failing later. A bad .config, missing Module.symvers, or wrong vermagic can otherwise turn into a vague modprobe failure several minutes downstream.

Installing the operator

The Helm chart is published as an OCI artifact on GHCR:

helm install vgpu-driver-operator \
  oci://ghcr.io/andrewreid/charts/vgpu-driver-operator \
  --version 2026.5.1 \
  -n vgpu-driver-operator \
  --create-namespace

For local development, installing from the repo works too:

helm install vgpu-driver-operator charts/vgpu-driver-operator \
  -n vgpu-driver-operator \
  --create-namespace

The chart installs the CRD, operator deployment, RBAC, and the build assets ConfigMap. It doesn’t magically know where your NVIDIA vGPU driver installers live, so you need to provide credentials separately.

For S3-compatible storage:

kubectl create secret generic s3-driver-storage-secret \
  -n vgpu-driver-operator \
  --from-literal=S3_ENDPOINT_URL=https://s3.example.com \
  --from-literal=AWS_ACCESS_KEY_ID=<access-key> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<secret-key>

For a private registry:

kubectl create secret docker-registry private-registry-secret \
  -n vgpu-driver-operator \
  --docker-server=registry.example.com \
  --docker-username=<username> \
  --docker-password=<password>

Then apply a VGPUDriverImage and watch it reconcile:

kubectl apply -f vgpu-driver-image.yaml
kubectl get vgpudriverimages -n vgpu-driver-operator -w
kubectl logs -n vgpu-driver-operator \
  -l app.kubernetes.io/name=vgpu-driver-operator -f

The operator status records what it discovered and what it built: observed Flatcar versions, tracked channel versions, build phases, job names, tags, and any garbage collection actions.

Wiring it back into the GPU Operator

Once the images exist in the registry, the NVIDIA GPU Operator can consume them. The two operators are deliberately decoupled: my operator builds and pushes images; NVIDIA’s operator runs them.

This is also where I lost a bit more time than I expected. The GPU Operator documentation has Flatcar notes, but I couldn’t get the documented path to work cleanly for my RKE2 cluster. The driver deployment and the container toolkit both needed a bit of Flatcar/RKE2-specific nudging.

The important bits in my working gpu-operator values ended up looking roughly like this:

driver:
  usePrecompiled: true
  repository: registry.k3s.wp.reid.ee
  image: vgpu-driver
  version: "535.261.03"
  imagePullPolicy: Always
  licensingConfig:
    configMapName: nvidia-vgpu-licensing-config
    nlsEnabled: true

toolkit:
  installDir: /opt/nvidia
  env:
    - name: CONTAINERD_SOCKET
      value: /run/k3s/containerd/containerd.sock

cdi:
  nriPluginEnabled: true

There’s a bit going on there.

First, the driver image is split across repository, image, and version, rather than being one full image string. With those values, the GPU Operator constructs the actual image reference itself. On my nodes, with usePrecompiled: true, that means it looks for a tag like:

registry.k3s.wp.reid.ee/vgpu-driver:535.261.03-6.12.81-flatcar-flatcar4593.2.0

That’s exactly the tag format the vgpu-driver-operator builds and pushes.

Second, the container toolkit needs to know where RKE2’s containerd socket actually lives. The defaults didn’t match my nodes. RKE2 uses the k3s-style path:

/run/k3s/containerd/containerd.sock

Without that, the toolkit daemonset came up but didn’t configure the runtime I was actually using. That’s the particularly annoying class of failure where the driver daemonset can be healthy, the device plugin can be present, and your workload still doesn’t get a useful NVIDIA runtime.

Third, I set the toolkit install directory to /opt/nvidia. Flatcar’s immutable layout means anything that assumes a conventional mutable Linux filesystem is immediately suspect. Keeping the NVIDIA bits under /opt/nvidia matched the shape the operator was happy with, and avoided trying to treat the host like a normal package-managed distribution.

Finally, I enabled the CDI/NRI path:

cdi:
  nriPluginEnabled: true

This is one of those settings that’s easy to skip when you’re reading examples for a different runtime or distro. In this setup, I wanted the toolkit to integrate with containerd cleanly rather than relying on a Docker-era runtime class mental model.

The full HelmRelease in my Flux repo also configures the device plugin for time-slicing, because the Tesla P40 is there for homelab workloads rather than one big exclusive training job:

devicePlugin:
  config:
    create: true
    name: nvidia-device-plugin-config
    default: time-slicing
    data:
      time-slicing: |-
        version: v1
        flags:
          migStrategy: none
        sharing:
          timeSlicing:
            resources:
              - name: nvidia.com/gpu
                replicas: 10

For a runtime-compiled image, the driver section would be the same idea but with usePrecompiled: false:

driver:
  repository: registry.k3s.wp.reid.ee
  image: vgpu-driver
  version: "535.261.03"
  usePrecompiled: false

For precompiled images, which is what I’m using, it’s the same thing with precompiled mode enabled:

driver:
  repository: registry.k3s.wp.reid.ee
  image: vgpu-driver
  version: "535.261.03"
  usePrecompiled: true

Then install or upgrade the GPU Operator in the usual way:

helm repo add nvidia https://nvidia.github.io/gpu-operator
helm repo update

helm upgrade --install gpu-operator nvidia/gpu-operator \
  -n gpu-operator \
  --create-namespace \
  -f gpu-operator-values.yaml

The important part is that driver.version remains just the NVIDIA driver version. The GPU Operator appends the Flatcar and kernel suffixes itself based on the node.

What this buys me

The big win is that Flatcar upgrades no longer require me to remember a manual driver-image dance.

When the cluster gets a new Flatcar version, the operator can discover it from the nodes or from the tracked release channel, notice that the matching NVIDIA driver image is missing, and start a build. When old Flatcar versions age out, the retention policy can prune registry tags.

This also keeps the responsibilities in the right places:

  • Flatcar developer images provide the build environment.
  • The vGPU driver operator builds and publishes driver images.
  • The NVIDIA GPU Operator manages runtime GPU integration.
  • RKE2 runs the cluster without host-level snowflake driver installs.

It took more supporting work than I expected, especially getting the Flatcar developer image into a reliable Docker workflow and matching NVIDIA’s tag conventions exactly. But the final shape is much nicer than a pile of scripts.

The useful pattern here isn’t limited to NVIDIA vGPU. If you run an immutable or minimal Kubernetes node OS, and something in your stack needs kernel-matched build artifacts, it’s worth separating the problem into:

  1. publish a versioned OS build environment;
  2. build the artifact in Kubernetes;
  3. tag it using the consumer’s native convention;
  4. let the runtime operator keep doing the runtime operator’s job.

Open questions

I have no idea if this is going to be of use to others, but it’s here if it does. I’m interested to know if this method will work for others, publishing precompiled drivers for different NVIDIA drivers etc for those using less ancient GPUs. I certainly intend to keep this working for my own purposes.

Could I have done this without the help of Codex and Claude? Probably not. I mean, possibly, but whether I’d have had the time and interest to dedicate to getting this working, in amongst everything else? Doubtful. To that end, while I don’t doubt the code qualtiy isn’t exactly going to win any awards, I don’t really care – it’s doing a useful job in my homelab and the automagicity of it all makes me feel good in my Kubernetes parts!