A container runs what it was last created with¶
This one applies well beyond the box it happened on, so I'm writing it down separately.
I had a swap script that started a container with docker start. I'd edited the compose file to point at a new image and a new model path, ran the swap, and it came up running the old pair. Nothing was wrong with the compose file. Nothing was wrong with the script. The container was simply doing what containers do.
docker start starts a container. It does not apply a compose file. A container comes up with whatever image and command it was created with. Editing a .yml on disk changes nothing until you recreate.
Warning
This is the kind of bug that reads like a config problem and is actually a lifecycle problem. You can stare at a correct compose file for an hour while a stale container runs.
The check that settles it in one line:
That tells you what a swap will actually run, which is not necessarily what the file says.
Staging costs nothing¶
A recreate is GPU-free — create makes the container and leaves it stopped. So you can stage a candidate without touching VRAM:
docker compose -f docker-compose.v3.yml create --force-recreate # stage v3, no GPU
docker compose create --force-recreate # stage v2 again, no GPU
Both pairs sat on disk with identical 23.7 GB artifacts, so switching between them was a recreate, not a copy.
Image ninfer:local, artifact qwen3_8_27b_nvfp4.ninfer.
Has actually run on the GPU. This is the fallback.
Image ninfer:v3, artifact qwen3_8_27b_nvfp4_v3.ninfer.
Built, verified with upstream's own reader — version 3, 1190 objects, exactly the count the upgrade script lists as correct — and never started on a GPU.
The rule I follow now: the deployed pair is the verified pair. A build that has never run on hardware is not a fallback, no matter how correct its metadata looks. The artifact format broke between v2 and v3, so a rebuild from the wrong tree produces an engine that can't read the model at all.
The same bug in a different shape¶
The compose-file version of this on another service: depends_on pointed at an image that wasn't on the box. Compose tried to pull it, got pull access denied, and aborted before recreating the service — so the command exited looking fine while the old container kept running.
Both cases are the same failure: a command that reports success without changing the running state. Check the running state, not the command's exit code.