9. Orchestration: running fleets, not machines

Orchestration is software that keeps a fleet of machines doing what you declared: placing work, restarting what fails, scaling up and down. Kubernetes dominates services and inference, Slurm dominates large training clusters, and Ray spreads Python code across many machines.

Why orchestration exists

With one server you SSH in and run things. With 50 servers and 300 containers you need software to decide where each container runs given its CPU, GPU and memory needs. It must also restart failures, roll out new versions without downtime, let services find each other, and scale.

The core idea is declarative desired state plus a reconciliation loop. You declare “I want three copies of this image, each with one GPU.” A controller constantly compares reality to that declaration and acts on the difference, like a thermostat. The same pattern appears in Terraform, GitOps and every autoscaler, so learn it once.

Kubernetes concepts

The control plane has four parts: the API server (the front door for every change), etcd (the database holding desired and actual state), the scheduler (assigns work to machines), and controllers (the reconciliation loops). Each worker node runs an agent called the kubelet plus a container runtime.

The objects you will use:

Each container states requests (what the scheduler reserves for it) and limits (what it may not exceed). GPUs are requested as whole devices with a line such as nvidia.com/gpu: 1, and are never overcommitted.

A minimal model-server Deployment, to make it concrete:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-server
spec:
  replicas: 2
  selector:
    matchLabels: {app: llm-server}
  template:
    metadata:
      labels: {app: llm-server}
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.x.y   # pin an exact version
          args: ["--model", "meta-llama/Llama-3.1-8B-Instruct"]
          ports: [{containerPort: 8000}]
          resources:
            limits: {nvidia.com/gpu: 1}

Managed Kubernetes (Amazon EKS, Google GKE, Azure AKS) runs the control plane for you; you manage groups of worker nodes. More automated modes (GKE Autopilot, EKS Auto Mode) also manage the nodes.

GPUs on Kubernetes

• Device plugin and GPU Operator. NVIDIA’s device plugin tells the scheduler how many GPUs each node has. The GPU Operator installs the driver, container toolkit, device plugin and a metrics exporter (DCGM) for you.
• Node pools per GPU type. Put H100s, L4s and CPU-only machines in separate pools. Add a taint to GPU nodes so only pods that explicitly tolerate it land there; otherwise a stray logging pod can block an expensive GPU.
• Two layers of autoscaling. Pod autoscalers (HPA, or KEDA for scaling on queue length) add pods; node autoscalers (Cluster Autoscaler, or Karpenter on AWS) add machines. A new GPU node takes minutes: provisioning, driver start, image pull, then model loading. Plan for that delay.
• Gang scheduling. Distributed training needs all of its pods running at once or none at all. The default scheduler places pods one by one, so a 16-pod job can grab 12 GPUs and wait forever. Kueue, Volcano or the Kubeflow Training Operator handle all-or-nothing scheduling and queues.
• Finer device sharing. Kubernetes’ newer Dynamic Resource Allocation (DRA) API allows richer GPU requests, such as specific slices or interconnect topology.

Slurm

Slurm is the batch scheduler from the supercomputing world, and still the standard on large training clusters. Many GPU clouds hand you a Slurm cluster by default.

• You write a short script stating nodes, GPUs per node and a time limit, then submit it with sbatch. Slurm queues the job, allocates whole nodes, and launches your processes with srun.
• Strengths: all-or-nothing allocation is built in, it understands network topology, and researchers find it simple.
• Weaknesses: it is built for jobs that finish, not for long-running services, and it fits less naturally with cloud-native tooling.
• Managed and hybrid options: AWS ParallelCluster and SageMaker HyperPod, Google’s Cluster Toolkit, and projects that run Slurm on Kubernetes.

Ray

Ray makes ordinary Python distributed. Decorate a function with @ray.remote and it becomes a task that can run on any machine in the cluster; decorate a class and it becomes a stateful actor.

On top sit libraries for common AI jobs: Ray Data (distributed processing and batch inference), Ray Train (distributed training), Ray Tune (hyperparameter search) and Ray Serve (model serving). Ray runs on plain VMs, on Kubernetes through KubeRay, or as a managed service from Anyscale. It is widely used for reinforcement-learning post-training pipelines and for jobs like embedding a billion documents.

Launchers and managed ML platforms

• SkyPilot (and the similar dstack): you describe what you need, such as “8 × H100,” and it finds the cheapest available capacity across clouds and regions, runs your job, and shuts everything down. Ideal for individuals and small teams chasing scarce GPUs.
• Managed ML platforms (Amazon SageMaker, Google Vertex AI, Azure Machine Learning) bundle training jobs, model endpoints, pipelines, notebooks and model registries, wired into the provider’s identity and storage. Faster to start, more lock-in, and sometimes a price premium per instance.

Choosing

Check yourself

• What is a reconciliation loop, and where else in the cloud have you now seen one?
• Why can a 16-pod training job get stuck forever on a default scheduler when only 12 GPUs are free?
• Why do GPU nodes get a taint, and what do GPU pods need to land there?
• The autoscaler adds a GPU node, yet the first request still waits several minutes. Where does that time go?