SkyPilot: Running ML and AI Workloads on Any Cloud

Training and serving machine learning models often means fighting two battles at once: finding GPUs that are actually available, and avoiding surprise cloud bills. SkyPilot is an open-source framework that sits above AWS, GCP, Azure, Kubernetes, and your own machines, letting you describe a job once and run it wherever it is cheapest and available. This tutorial walks through the concepts, the YAML format, the CLI, managed spot jobs, and serving — with practical examples you can adapt.

What SkyPilot Is and the Problems It Solves

SkyPilot is a layer that provisions and manages compute on whichever backend you choose, then runs your workload on it. You write a declarative description of what you need (an accelerator, some CPUs, a setup script, a run command) and SkyPilot figures out where to place it.

It addresses a handful of recurring pain points in ML infrastructure:

GPU scarcity. A specific GPU type is frequently out of capacity in one region while available in another. SkyPilot searches across regions and clouds to find capacity instead of leaving you to retry manually.
Cost. The same accelerator can vary widely in price between providers and regions. SkyPilot's optimizer compares options and picks the cheapest one that satisfies your constraints.
Multi-cloud lock-in. Teams often standardize on one provider not because it is best, but because rewriting infrastructure is painful. SkyPilot uses the same task definition everywhere, so moving a job from AWS to GCP is a flag change, not a rewrite.
Manual cluster management. Spinning up VMs, installing drivers, syncing code, and remembering to tear everything down is tedious and error-prone. SkyPilot automates provisioning, code sync, and — importantly — auto-stopping idle clusters so you stop paying for forgotten machines.

It is not a replacement for your cloud account; it uses your existing credentials and your own quota. Think of it as an orchestration layer rather than a managed hosting service.

How It Works

At a high level, SkyPilot does the following when you launch a task:

Reads your task definition (a YAML file or Python call) describing resources and commands.

Runs the optimizer, which queries pricing and availability across the clouds you have enabled, then ranks candidate (cloud, region, instance type) combinations.

Provisions a cluster on the winning option using your cloud credentials, retrying other regions or clouds if capacity is unavailable.

Syncs your local working directory and any mounted files to the cluster.

Runs your setup commands once, then your run command.

Streams logs back to your terminal and, if configured, auto-stops the cluster after a period of inactivity.

The cluster is a normal set of VMs (or Kubernetes pods) in your account. You can SSH into it, run more commands on it, stop it, or tear it down at any time.

Installation and Cloud Setup

SkyPilot installs with pip. You select which backends to enable through extras:

pip install "skypilot[aws,gcp,kubernetes]"

Add or remove extras to match your environment — for example skypilot[azure], skypilot[lambda], or skypilot[all]. It is good practice to install into a dedicated virtual environment:

python -m venv .venv source .venv/bin/activate pip install "skypilot[aws,gcp,kubernetes]"

Next, configure credentials for each cloud you intend to use. SkyPilot relies on each provider's standard credential mechanism, so if your CLI already works, SkyPilot usually does too.

# AWS: configure access keys or SSO aws configure GCP: authenticate and set a project gcloud auth login gcloud auth application-default login gcloud config set project YOURPROJECTID Kubernetes: point SkyPilot at a working kubeconfig context kubectl config current-context

Finally, verify what SkyPilot can see with sky check:

sky check

This reports each cloud as enabled or disabled and explains what is missing if a cloud is not ready (for example, missing credentials or insufficient permissions). Only enabled clouds participate in the optimizer.

The Task YAML: The Central Concept

Everything in SkyPilot revolves around a task definition. The most common form is a YAML file. Here is an annotated example that covers the main fields:

# train.yaml
resources:
  accelerators: A100:1      # one A100 GPU; use A100:8 for eight
  cloud: aws                # optional: pin to a specific cloud
  cpus: 8+                  # at least 8 vCPUs
  memory: 32+               # at least 32 GB RAM
  disksize: 256            # boot/data disk in GB

  ports: 8888               # open a port (e.g., for a notebook)
  usespot: false           # set true to request spot/preemptible

workdir: .                  # sync the current directory to the cluster

filemounts:

  /data: s3://my-bucket/datasets   # mount a bucket as a path
  ~/.netrc: ~/.netrc               # copy a local file to the cluster

envs:
  WANDBAPIKEY: ""         # declared here, value supplied at launch

  EPOCHS: "10"

setup: |
  pip install -r requirements.txt

run: |
  python train.py --epochs $EPOCHS --data-dir /data

The key fields:

resources describes the hardware. accelerators uses the NAME:COUNT form (A100:1, T4:4, H100:8). You can also leave cloud unset to let the optimizer choose, and set usespot: true to request discounted preemptible instances.
setup runs once when the cluster is first provisioned (or when the setup changes). Put dependency installation here so repeated runs are fast.
run is the actual command that executes your workload.
workdir syncs a local directory to the cluster before each run. Keep it small; large datasets belong in filemounts.

filemounts maps remote paths to local files or cloud buckets. Mounting a bucket gives the job direct access to object storage without copying everything up front.
envs declares environment variables. Declaring them with empty values lets you pass real values securely at launch time.

You can specify multiple accelerator candidates so the optimizer has more room to find capacity:

resources:
  accelerators: {A100:1, A10G:1, L4:1}
  usespot: true

Core CLI

A handful of commands cover day-to-day work.

Launch a task onto a named cluster (creating it if needed):

sky launch train.yaml -c mycluster --env WANDBAPIKEY=$WANDBAPIKEY

The -c flag names the cluster so you can reuse it. SkyPilot prints the optimizer's plan and asks for confirmation before provisioning.

Check the state of your clusters:

sky status

This lists each cluster, its cloud and region, the instance type, status (UP, STOPPED, INIT), and the autostop setting.

Inspect the job queue on a cluster and stream logs:

sky queue mycluster sky logs mycluster # follow the latest job sky logs mycluster 2 # a specific job by ID

Run an additional command on a cluster that already exists, reusing the provisioned hardware (and skipping setup if unchanged):

sky exec mycluster -- python evaluate.py --checkpoint /data/ckpt.pt

Manage the cluster lifecycle:

sky stop mycluster # stop (keep the disk; resume later, billed for storage) sky start mycluster # restart a stopped cluster sky down mycluster # terminate and delete everything

Set an autostop so an idle cluster shuts itself down:

sky autostop mycluster -i 10 # stop after 10 minutes idle or at launch time: sky launch train.yaml -c mycluster -i 10

Use --down instead of -i if you want the cluster fully terminated (not just stopped) after idling.

The Optimizer

When you do not pin a cloud, SkyPilot's optimizer evaluates every enabled (cloud, region, instance) combination that satisfies your resources block, factoring in current pricing and known availability. It then prints a table and selects the cheapest viable option.

A launch might show something like this:

Considered resources (1 node): CLOUD INSTANCE vCPUs Mem(GB) ACCELERATORS REGION COST ($/hr) GCP a2-highgpu-1g 12 85 A100:1 us-central1 3.67 AWS p4d.24xlarge 96 1152 A100:8 us-east-1 32.77 Azure StandardNC24ads 24 220 A100:1 eastus 3.91The mark indicates the selected (cheapest) option.

If provisioning the selected option fails because of capacity, SkyPilot automatically falls back to the next candidate region or cloud. This failover behavior is one of the main reasons it handles GPU scarcity better than launching VMs by hand.

Managed Jobs and Spot Recovery

For long-running training, sky launch ties the job to your terminal session. Managed Jobs run the job under a controller that supervises it, so you can disconnect and the job keeps going:

sky jobs launch train.yaml -n my-training-run

Monitor and inspect managed jobs:

sky jobs queue sky jobs logs --name my-training-run sky jobs cancel my-training-run

The combination of managed jobs and spot instances is a notable feature. Spot (preemptible) instances are much cheaper but can be reclaimed by the cloud at any time. With a managed spot job, SkyPilot watches for preemption: when an instance is reclaimed, it automatically provisions a replacement — possibly in a different region or cloud — and resumes the job.

# train-spot.yaml resources: accelerators: A100:1 usespot: true filemounts: /checkpoints: s3://my-bucket/checkpoints setup: | pip install -r requirements.txt run: | # Resume from the latest checkpoint if one exists python train.py \ --checkpoint-dir /checkpoints \ --resume-if-exists

sky jobs launch train-spot.yaml -n spot-training

The key to making this work is checkpointing. SkyPilot handles re-provisioning, but your code must periodically save progress to durable storage (a mounted bucket) and resume from it on restart. With that in place, a job interrupted halfway picks up near where it left off rather than starting over. This is what makes spot economics practical for multi-hour training runs.

SkyServe: Serving Models with Autoscaling

SkyServe turns a task into a service with a stable endpoint, health checks, autoscaling, and replicas that can span regions. You add a service section to the YAML describing the readiness probe and replica policy.

# serve-llm.yaml
service:
  readinessprobe:

    path: /v1/models      # health check endpoint
  replicas: 2             # or use a min/max range to autoscale

resources:
  accelerators: {L4:1, A10G:1}
  ports: 8000
  usespot: true

setup: |
  pip install vllm

run: |
  python -m vllm.entrypoints.openai.apiserver \

    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --port 8000

Bring the service up and manage it:

sky serve up serve-llm.yaml -n my-llm sky serve status my-llm sky serve logs my-llm 1 sky serve down my-llm

SkyServe gives you a single endpoint that load-balances across replicas. Because replicas can run on spot across multiple regions, you get cost savings and resilience: if one region loses capacity, traffic shifts to healthy replicas while a replacement spins up.

Running on Kubernetes

If you have a Kubernetes cluster with GPUs (in the cloud or on-prem), SkyPilot treats it as just another backend. Once your kubeconfig context is reachable and sky check shows Kubernetes as enabled, you pin a task to it like any other cloud:

resources: accelerators: A100:1 cloud: kubernetes

sky launch train.yaml -c k8s-run

SkyPilot launches pods, syncs your code, and runs the job using the same YAML and CLI you use for VMs. This lets a team run on an internal Kubernetes cluster when capacity exists there and burst to a public cloud when it does not — without changing the task definition.

Syncing Data with Object Storage

For datasets and outputs, mounting cloud object storage is usually better than copying via workdir. SkyPilot Storage and filemounts support S3, GCS, and other object stores:

filemounts:
  # Mount an existing bucket read-only for input data
  /data:
    source: s3://my-bucket/datasets
    mode: MOUNT

  # Create/sync a bucket for outputs, written back to the cloud
  /outputs:
    name: my-training-outputs
    store: gcs
    mode: MOUNT

In MOUNT mode, the bucket appears as a normal directory and reads/writes stream to object storage. This keeps large data off the boot disk and makes the same data available no matter which region or cloud the job lands on — which is exactly what spot recovery and the optimizer need.

End-to-End Example

Here is a coherent flow: define a GPU training job, launch it on the cheapest available cloud, then convert it to a managed spot job for the long run.

First, the task definition that lets the optimizer choose hardware and cloud:

# resnet.yaml resources: accelerators: {A100:1, A10G:1, L4:1} cpus: 8+disksize: 256 workdir: . filemounts: /data: source: s3://my-bucket/imagenet mode: MOUNT /checkpoints: name: resnet-checkpoints store: s3 mode: MOUNT envs: EPOCHS: "90" setup: | pip install -r requirements.txt run: | python trainresnet.py \ --data-dir /data \ --checkpoint-dir /checkpoints \ --resume-if-exists \ --epochs $EPOCHS

Launch interactively on the cheapest option, with a 15-minute autostop so it cleans up if you walk away:

sky launch resnet.yaml -c resnet-dev -i 15 sky logs resnet-dev

Once the code is verified, run the full job as a managed spot job so it survives preemptions and your laptop disconnecting:

sky jobs launch resnet.yaml -n resnet-full --use-spot sky jobs queue sky jobs logs --name resnet-full

When training finishes, the checkpoints are already in S3, and the managed job's resources are released automatically. If you used an interactive cluster, tear it down:

sky down resnet-dev

The Python SDK

If you prefer driving SkyPilot from code — for example inside a pipeline or experiment launcher — the Python SDK mirrors the CLI:

import sky

task = sky.Task(
    setup="pip install -r requirements.txt",
    run="python train.py --epochs $EPOCHS",
    workdir=".",
    envs={"EPOCHS": "10"},
)
task.setresources(

    sky.Resources(accelerators="A100:1", usespot=True)
)

Launch onto a named cluster
sky.launch(task, clustername="sdk-run", idleminutestoautostop=10)

Or submit as a managed job
sky.jobs.launch(task, name="sdk-managed")

The SDK is useful when you need to generate tasks programmatically (sweeps, parameterized runs) or integrate SkyPilot into an existing orchestration layer.

Cost-Control Best Practices

The flexibility SkyPilot provides only saves money if you use it deliberately. A few habits matter most.

Always set an autostop or autodown. The single biggest source of waste is idle GPU clusters. Use -i for development clusters and --down for one-off runs so nothing lingers.
Use spot for fault-tolerant work. Training with checkpointing, batch inference, and data processing are good fits for spot. Reserve on-demand for latency-sensitive serving that cannot tolerate interruptions, or use a small on-demand fallback alongside spot replicas.
Right-size resources. Request the smallest accelerator that meets your throughput needs, and offer the optimizer a set of candidates (for example {A100:1, A10G:1, L4:1}) rather than pinning one type. More candidates means better odds of finding cheap capacity.
Let the optimizer pick the cloud. Leaving cloud unset gives SkyPilot the freedom to find the cheapest available region across all enabled providers.
Keep data in object storage, not on disk. Mounting buckets avoids re-uploading data on every run and keeps boot disks small, which also makes spot recovery and cross-region placement work cleanly.
Prefer managed jobs for anything long. They survive disconnects and preemptions, so you are not paying for a failed half-finished run.

Conclusion and Key Takeaways

SkyPilot reduces the friction of running ML and AI workloads across clouds by turning "where and how do I provision this" into a declarative task definition plus a small set of commands.

A single task YAML (resources, setup, run, workdir, file_mounts, envs) describes a job once and runs it anywhere.
The optimizer compares clouds and regions and picks the cheapest available option, with automatic failover when capacity is short.
Managed jobs with spot recovery make discounted, preemptible compute practical for long training runs, provided your code checkpoints to durable storage.
SkyServe extends the same model to autoscaling, multi-region inference endpoints.
Kubernetes is just another backend, so on-prem and cloud capacity share one workflow.
Disciplined use of autostop, spot, right-sizing, and object storage is where the real cost savings come from.

The practical path is to start small: enable one or two clouds, run a short job interactively with a short autostop, then graduate the workload to a managed spot job once it is verified. From there, the same definitions extend naturally to serving and to additional backends as your needs grow.

SkyPilot Tutorial: Running ML and AI Workloads on Any Cloud

SkyPilot: Running ML and AI Workloads on Any Cloud

What SkyPilot Is and the Problems It Solves

How It Works

Installation and Cloud Setup

GCP: authenticate and set a project

Kubernetes: point SkyPilot at a working kubeconfig context

The Task YAML: The Central Concept

Core CLI

or at launch time:

The Optimizer

Managed Jobs and Spot Recovery

SkyServe: Serving Models with Autoscaling

Running on Kubernetes

Syncing Data with Object Storage

End-to-End Example

The Python SDK

Launch onto a named cluster

Or submit as a managed job

Cost-Control Best Practices

Conclusion and Key Takeaways

Related Articles

Complete Terraform for ML Tutorial: Infrastructure as Code for Machine Learning

NVIDIA NIM: A Complete Guide to NVIDIA Inference Microservices

Complete Prompt Flow Tutorial: Microsoft Framework for Building LLM Applications

Truss: How to Package, Serve, and Deploy ML Models Cleanly

Related Articles

Complete Terraform for ML Tutorial: Infrastructure as Code for Machine Learning

Tutorial Lengkap Terraform untuk ML: Infrastructure as Code untuk Machine Learning Terraform adalah tool Infrastructure ...

NVIDIA NIM: A Complete Guide to NVIDIA Inference Microservices

Tutorial Lengkap NVIDIA NIM (NVIDIA Inference Microservices) untuk Developer Indonesia Penulis: Ruby Abdullah Tanggal: 2...

Complete Prompt Flow Tutorial: Microsoft Framework for Building LLM Applications

Tutorial Lengkap Prompt Flow: Framework Microsoft untuk Membangun Aplikasi LLM Prompt Flow adalah framework open-source ...

Truss: How to Package, Serve, and Deploy ML Models Cleanly

Truss: Cara Package, Serve, dan Deploy Model ML dengan Rapi Halo temen-temen, ketemu lagi sama aku, Ruby. Kali ini aku p...