9. Docker Compose for multi-service AI apps

Compose replaces a pile of docker run commands with one declarative YAML file, so an AI application made of several services (an API, a model server, a vector database, a cache, a background worker) starts with docker compose up. It runs on a single host; for several machines you move to Kubernetes (Part 15).

The concepts

• Project. One Compose file and its directory; the project name prefixes every container, network and volume it creates.
• Service. One container definition: image or build, command, environment, ports, volumes, limits. The service name is also its DNS name on the project network.
• Network. Compose creates one network per project automatically, so services reach each other by name with no extra setup.
• Volume. Named volumes declared at the bottom of the file persist across down and up.
• Profiles and override files. Ways to switch services or settings on and off for different environments.

A RAG stack that runs on your Mac and on a GPU server

The base file defines everything except the LLM. The LLM’s address comes from an environment variable, so the same stack can call Ollama on your Mac or a vLLM container on a GPU server.

compose.yaml:

services:
  api:
    build: .
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      LLM_BASE_URL: ${LLM_BASE_URL}
      DATABASE_URL: postgresql://rag:rag@db:5432/rag
      REDIS_URL: redis://redis:6379/0
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
    restart: unless-stopped

  worker:
    build: .
    command: ["python", "-m", "src.worker"]
    environment:
      LLM_BASE_URL: ${LLM_BASE_URL}
      DATABASE_URL: postgresql://rag:rag@db:5432/rag
      REDIS_URL: redis://redis:6379/0
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy

  db:
    image: pgvector/pgvector:pg16
    environment:
      POSTGRES_USER: rag
      POSTGRES_PASSWORD: rag   # development only
      POSTGRES_DB: rag
    volumes:
      - pg-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U rag -d rag"]
      interval: 5s
      timeout: 3s
      retries: 10

  redis:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 10

volumes:
  pg-data:

compose.gpu.yaml, added only on the GPU server:

services:
  llm:
    image: vllm/vllm-openai:latest   # pin an exact version in practice
    command: ["--model", "Qwen/Qwen2.5-7B-Instruct", "--max-model-len", "8192"]
    environment:
      HF_TOKEN: ${HF_TOKEN}
    volumes:
      - hf-cache:/root/.cache/huggingface
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  hf-cache:

How you run it:

• On your Mac, run Ollama natively (it uses the Apple GPU through Metal), put LLM_BASE_URL=http://host.docker.internal:11434/v1 in .env, and run docker compose up -d.
• On the GPU server, put LLM_BASE_URL=http://llm:8000/v1 and HF_TOKEN=… in .env, and run docker compose -f compose.yaml -f compose.gpu.yaml up -d. Later files are merged over earlier ones.

Notice that both api and llm listen on port 8000 without conflict: each has its own network namespace. Only published ports share the host’s port space.

What the file is teaching you

• ${VAR} interpolation. Compose reads a .env file in the project directory automatically and substitutes its values into the YAML. Keep .env out of Git. docker compose config prints the fully resolved file, which is the first thing to check when a value seems wrong.
• depends_on with conditions. service_healthy waits until the dependency’s healthcheck passes; plain depends_on only waits for the container to start, not for the service inside to be ready. Your code should still retry connections, because services can restart later. The API doesn’t depend on llm here: a model can take minutes to load, so the API should handle the LLM being briefly unavailable.
• ipc: host gives the model server the host’s shared memory, which vLLM and PyTorch multi-process code need (Part 10). shm_size: 16gb is the more contained alternative.
• The deploy.resources.reservations.devices block is Compose’s way of saying --gpus all.
• Profiles (profiles: ["eval"] on a service) keep optional services, such as an evaluation runner or a Jupyter container, out of the default up until you add --profile eval.

Everyday commands

For development, bind-mount ./src into the container and run the server with auto-reload, or use docker compose watch, which syncs changed files into running containers and rebuilds when dependency files change.