1. The problem Docker solves

Your code never runs on “Python”; it runs on a precise stack of interpreter, libraries, system packages, kernel and drivers. Docker packages everything above the kernel so that stack travels with the code, and leaves the kernel and the GPU driver to the host. That boundary is the single most important fact for AI work.

The stack your code actually depends on

A train.py that imports PyTorch depends, from top to bottom, on: your source files; pinned Python packages with compiled binaries inside them; a specific Python interpreter and its ABI; system libraries such as glibc, libgomp or ffmpeg; CUDA user-space libraries (often bundled inside the PyTorch wheel); the Linux kernel; the NVIDIA kernel driver; and the hardware. Change any one of these and behaviour can change. The diagram below shows which of these the image carries.

[embedded content: what the image carries vs what the host provides · 8 layers]

Everything above the dashed line ships inside the image; everything below must already be right on the machine. That’s why a too-old driver or the wrong CPU architecture breaks even a perfectly built image.

Why machine learning makes this worse

• Compiled extensions. torch, flash-attn, bitsandbytes and similar packages ship machine code built against specific CUDA, glibc and CPU-architecture versions. A mismatch shows up as import errors, segfaults or missing GPU kernels.
• Driver coupling. CUDA libraries need a host driver at least as new as they expect. Your laptop, a university cluster and a cloud VM usually differ.
• Dependency rot. Research code with unpinned requirements often fails to install a few months later, because new releases of transitive dependencies break it.
• Everything changes at once. Moving from a macOS arm64 laptop to a Linux x86_64 GPU server to a Kubernetes cluster changes OS, CPU architecture and GPU in one step.

Where Docker sits among the alternatives

Tool	What it isolates	What it shares with the host	Typical use
venv, uv	Python packages only	Interpreter, system libraries, kernel, driver	Local development
conda, mamba	Python packages plus some native libraries	Kernel, driver, most system libraries	Scientific stacks with native deps
Container (Docker, Podman)	Whole user space: files, libraries, processes, network view	Kernel and drivers	Shipping and running apps and jobs
Virtual machine	Everything, including its own kernel	Only the hardware (via a hypervisor)	Strong isolation, other operating systems
Nix, Guix	Exact build of every dependency	Kernel, driver	Bit-for-bit reproducible builds

What Docker does not solve

• Kernel and driver differences. A container on a host with an old NVIDIA driver still fails; the driver is not in the image.
• CPU architecture. An arm64 image does not run natively on an amd64 server, and the reverse. Emulation works but is slow.
• Hardware numerics. Different GPUs give slightly different floating-point results, so bit-identical training across hardware is not guaranteed.
• Build reproducibility. A Dockerfile run next month can produce different bytes, because apt-get and pip fetch whatever is current. What is reproducible is the built image: once pushed, its digest identifies the exact bytes forever.

The practical consequence for research: archive the image digest alongside your results, not just the Dockerfile. The Dockerfile tells people how you built it; the digest lets them run exactly what you ran.