5. Container lifecycle

A container lives exactly as long as its main process, PID 1. Starting, stopping, restarting, logging and exit codes all follow from that one rule.

States and the commands that move between them

A container goes from created to running, can be paused, ends up exited, and disappears when removed. docker create makes one without starting it; docker start starts it; docker run pulls the image if needed, creates and starts in one go. docker stop asks the process to finish, docker kill forces it, and docker rm deletes the container and its writable layer.

docker exec is different: it starts an extra process inside an already running container, in the same namespaces and cgroup. It’s how you get a shell for debugging (docker exec -it api bash). It does not create a new container, and the container still stops when PID 1 exits, even if your exec shell is open.

Anatomy of a docker run

docker run -d --name llm-api \
  -p 127.0.0.1:8000:8000 \
  --env-file .env -e MODEL_ID=Qwen/Qwen2.5-7B-Instruct \
  -v hf-cache:/cache/hf \
  --gpus all --shm-size 8g \
  --memory 32g --cpus 8 \
  --restart unless-stopped \
  ghcr.io/ulvi/llm-api:1.2.0 --port 8000

PID 1 and signals

docker stop sends SIGTERM to PID 1, waits 10 seconds (change it with -t or Compose’s stop_grace_period), then sends SIGKILL. Linux treats PID 1 specially: signals it hasn’t registered a handler for are ignored rather than killing it. So a process that doesn’t handle SIGTERM sits there for 10 seconds and then gets killed hard.

The classic way to break shutdown is a shell-form command. CMD python serve.py actually runs /bin/sh -c "python serve.py", which makes the shell PID 1; many shells don’t forward SIGTERM to the child, so your server never hears it. Use exec form, CMD ["python", "serve.py"], end entrypoint scripts with exec "$@", or add --init to run a tiny init (tini) as PID 1 that forwards signals and reaps zombie processes.

For ML this is practical, not cosmetic. A graceful SIGTERM lets an inference server finish in-flight requests, and lets a training job save a checkpoint before a spot or preemptible VM is reclaimed; cloud providers send SIGTERM first.

Exit codes

To tell an OOM kill from a timeout, run docker inspect -f '{{.State.OOMKilled}}' <name>.

Restart policies

--restart no is the default. on-failure[:N] restarts only on a non-zero exit, optionally up to N times, which suits batch jobs. always and unless-stopped keep services up across crashes and daemon restarts; unless-stopped respects a manual docker stop. A container that crashes on start under always becomes a restart loop: check docker ps for a “Restarting” status and read the logs.

Logs

Docker captures whatever PID 1 writes to stdout and stderr; docker logs -f --tail 200 <name> follows it. Two traps for Python: output is block-buffered when not attached to a terminal, so set PYTHONUNBUFFERED=1 or logs arrive late; and the default json-file log driver never rotates unless configured, so a chatty week-long training run can fill the disk. Set --log-opt max-size=50m --log-opt max-file=3, or configure it once in the daemon settings.

Containers are disposable

Treat containers as replaceable: anything written outside a volume disappears with docker rm. docker commit can snapshot a running container into an image, but the result has no recipe and can’t be rebuilt; change the Dockerfile instead.