When the host is the dependency problem
A vLLM inference workload that would not run on RHEL 9, and why the fix was to stop trying to make it.
- vLLM
- Docker
- RHEL 9
- CUDA
- Inference
The requirement was ordinary enough: serve a few open-weight models to an internal application, on hardware we already had, on the distribution the rest of the fleet ran. The distribution was RHEL 9. The workload was vLLM. Those two facts turned a deployment into a week.
This is a write-up of what actually went wrong, because the interesting part is not the fix — the fix is a handful of lines — but why the obvious approaches all failed first.
The symptom
The install succeeds and then the import fails, or the import succeeds and it dies when it first touches the GPU. Both look like a Python problem and neither is. The traceback points at a wheel; the cause is underneath it.
Packages in this space are not really Python packages. They are compiled artefacts with a hard expectation about the C library, the CUDA runtime and the driver they were built against. A wheel that resolves cleanly can still be the wrong wheel for the machine it landed on, and the resolver has no way to tell you that.
Why an enterprise distribution makes it worse
RHEL's value is a stable ABI held still for years. That is exactly what you want underneath a database, and exactly what fights you here — the versions the published wheels target move much faster than the distribution does.
Which leaves two bad options. Take the distribution's Python and find the wheels want a newer toolchain; or install a newer toolchain alongside it and start maintaining a second, unsupported stack on a machine whose entire purpose was being supported.
What I tried that did not work
The common thread is that all three treat the host as fixed and the workload as the thing to bend. Every hour spent there was an hour spent making one specific machine special.
- Pinning versions until the resolver stopped complaining. It stopped complaining and it still did not run. A satisfied resolver is not the same as a binary that loads.
- Building from source on the host. That works — once, on that host. Then it is a machine nobody can reproduce, which is the thing I was trying to avoid.
- Layering a newer toolchain over the distribution's. Two package managers with an opinion about the same shared libraries is a problem you get to keep.
The fix
Stop negotiating with the host. Build on the base the upstream project actually targets, and let the host contribute nothing but a kernel and a driver.
# The base is chosen to match what the wheels were built against,
# NOT to match the rest of the fleet. That is the whole decision.
FROM ubuntu:22.04
# The CUDA runtime comes from the image. The host supplies the
# DRIVER only — that split is what makes the image portable.
RUN apt-get update \
&& apt-get install -y --no-install-recommends python3 python3-pip \
&& apt-get clean
RUN pip install --no-cache-dir vllm==<pinned>
# Explicit, not default: two models resident on one card will each
# happily reserve most of it, and the second load is the one that fails.
ENV VLLM_GPU_MEMORY_UTILIZATION=0.45What it bought, beyond working
The same image now runs on the original hardware, on a different host, and on cloud compute, because the only thing it asks of any of them is a driver. That is the difference between a deployment and a machine somebody fixed once.
It also forced the GPU memory question into the open. Left at its default, vLLM sizes its cache against whatever it finds free — fine for one model, and a problem the moment a second lands on the same card. Pinning the fraction per container turns an intermittent out-of-memory failure into arithmetic you can do before deploying.
What I would do differently
I spent too long on the first approach. The signal was there early — a resolver that succeeds while the binary fails to load is almost always an ABI mismatch — and I read it as something to pin around rather than something to containerise away.
The generalisable version: when a workload's dependencies are compiled artefacts, the base image is not a packaging detail. It is the dependency.