Modal: Serverless GPU Cloud for ML Model Deployment

One of the biggest challenges in machine learning isn't building the model, it's deploying it to production. You need to manage servers, configure GPU drivers, set up auto-scaling, and much more. Modal exists to eliminate all that complexity.

Modal is a serverless cloud platform that lets you run Python code in the cloud with GPU access, without managing any infrastructure. Just write a regular Python function, add a decorator, and Modal handles the rest.

Why Modal?

Comparison with the traditional approach:

• Set up EC2/GCE server with GPU
• Install CUDA, cuDNN, drivers
• Configure Docker, Kubernetes
• Set up load balancer, auto-scaling
• Monitoring, logging, alerting
• Total setup time: days to weeks

• Write a Python function
• Add the @app.function(gpu="A100") decorator
• Deploy with modal deploy
• Total time: minutes

Modal offers pay-per-second pricing, meaning you only pay when your code is actually running. No idle costs.

Installation and Authentication

Install Modal

pip install modal

Setup Authentication

modal setup

This command will open your browser for login and automatically configure the API token. After completion, you can start using Modal from the terminal right away.

Verify the installation:

modal --version

Basic Functions with @app.function

The core concept of Modal is simple: you define regular Python functions and mark them with decorators to run in the cloud.

import modal

app = modal.App("hello-modal")

@app.function()
def hello(name: str) -> str:
    return f"Hello, {name}! This is running on Modal cloud."

@app.localentrypoint()

def main():
    # Call the function running in the cloud
    result = hello.remote("World")
    print(result)

modal run helloapp.py

The hello function executes in the Modal cloud, not on your local machine. The @app.localentrypoint() decorator marks the function that runs locally as the entry point.

import modal

app = modal.App("gpu-example")

Using NVIDIA T4 (budget-friendly, inference)
@app.function(gpu="T4")
def inferencet4():
    import torch
    print(f"GPU available: {torch.cuda.isavailable()}")

    print(f"GPU name: {torch.cuda.getdevicename(0)}")

    return "T4 inference complete"

Using NVIDIA A10G (balanced performance/cost)
@app.function(gpu="A10G")
def traina10g():
    import torch
    print(f"GPU: {torch.cuda.getdevicename(0)}")
    print(f"Memory: {torch.cuda.getdeviceproperties(0).totalmem / 1e9:.1f} GB")

    return "A10G training complete"

Using NVIDIA A100 (high-performance training)
@app.function(gpu="A100")
def traina100():
    import torch
    print(f"GPU: {torch.cuda.getdevicename(0)}")
    return "A100 training complete"

Using NVIDIA H100 (cutting-edge, LLM training)
@app.function(gpu="H100")
def trainh100():

    import torch
    print(f"GPU: {torch.cuda.getdevicename(0)}")

    return "H100 training complete"

Multiple GPUs
@app.function(gpu=modal.gpu.A100(count=2))
def multigputraining():

    import torch
    print(f"Number of GPUs: {torch.cuda.devicecount()}")
    return "Multi-GPU training complete"

Choose GPUs based on your needs:

• T4: Inference, small-medium models, most cost-effective
• A10G: Balance between performance and cost, fine-tuning
• A100: Large model training, LLM fine-tuning
• H100: Highest performance, large-scale LLM training

Container Images (modal.Image)

Modal uses containers to run your code. You can customize the environment with modal.Image:

import modal

Basic image with pip packages
image = modal.Image.debianslim(pythonversion="3.11").pipinstall(

    "torch",
    "transformers",
    "accelerate",
    "scipy"
)

app = modal.App("custom-image", image=image)

@app.function(gpu="T4")
def runmodel():
    from transformers import pipeline
    classifier = pipeline("sentiment-analysis")
    result = classifier("Modal is amazing for ML deployment!")
    return result

Image with Conda

image = modal.Image.fromregistry(
    "continuumio/miniconda3"
).condainstall(
    "pytorch",
    "torchvision",
    channels=["pytorch", "nvidia"]
).pipinstall(

    "transformers"
)

image = modal.Image.fromdockerfile("./Dockerfile")

Image with Pre-built Model

def downloadmodel():
    from transformers import AutoModelForCausalLM, AutoTokenizer
    modelname = "microsoft/DialoGPT-medium"
    AutoTokenizer.frompretrained(modelname)
    AutoModelForCausalLM.frompretrained(modelname)

image = (
    modal.Image.debianslim(pythonversion="3.11")
    .pipinstall("torch", "transformers")

    .runfunction(downloadmodel)

)

With .runfunction(), the model is downloaded during image build, not at runtime. This makes cold starts much faster.

Volumes and Storage

Modal provides several storage options for persistent data:

Modal Volume (Persistent Storage)

import modal

app = modal.App("volume-example")
volume = modal.Volume.fromname("my-data-volume", createifmissing=True)


@app.function(volumes={"/data": volume})
def savedata():
    # Write data to volume
    with open("/data/results.txt", "w") as f:
        f.write("Training results: accuracy=0.95")
    volume.commit()  # Ensure data is persisted
    return "Data saved"

@app.function(volumes={"/data": volume})
def loaddata():

    volume.reload()  # Refresh to see latest changes
    with open("/data/results.txt", "r") as f:
        content = f.read()
    return content

nfs = modal.NetworkFileSystem.fromname("my-nfs", createifmissing=True)

@app.function(networkfilesystems={"/shared": nfs})
def sharedstorage():

    # NFS is suitable for sharing data between concurrently running functions
    with open("/shared/sharedconfig.json", "w") as f:
        import json
        json.dump({"batchsize": 32, "lr": 0.001}, f)

import modal

app = modal.App("web-api")

@app.function()
@modal.webendpoint(method="GET")
def health():
    return {"status": "healthy", "service": "ml-api"}

@app.function()
@modal.webendpoint(method="POST")

def predict(data: dict):
    text = data.get("text", "")
    # Perform prediction
    return {"prediction": "positive", "confidence": 0.95, "text": text}

import modal
from fastapi import FastAPI

webapp = FastAPI()
app = modal.App("fastapi-modal")

image = modal.Image.debianslim().pipinstall("fastapi")

@webapp.get("/")

async def root():
    return {"message": "Hello from Modal + FastAPI"}

@webapp.post("/predict")
async def predict(data: dict):
    return {"result": "prediction here"}

@app.function(image=image)
@modal.asgiapp()

def fastapiapp():
    return webapp

modal deploy webapp.py
Output: https://your-app--fastapi-app.modal.run

Scheduled Jobs (Cron)

Modal supports scheduling functions like cron jobs:

import modal
from datetime import datetime

app = modal.App("scheduled-jobs")

@app.function(schedule=modal.Period(hours=1))
def hourlydatarefresh():
    """Runs every hour to refresh data."""
    print(f"Refreshing data at {datetime.now()}")
    # Perform ETL, retrain model, etc.

@app.function(schedule=modal.Cron("0 6   "))

def dailymodelevaluation():
    """Runs daily at 6 AM UTC."""
    print(f"Running daily model evaluation at {datetime.now()}")
    # Evaluate model performance
    # Send alert if degradation detected

@app.function(schedule=modal.Cron("0 0   0"))

def weeklyretrain():

    """Runs every Sunday."""
    print(f"Starting weekly model retrain at {datetime.now()}")

# Set secrets via CLI
modal secret create my-api-keys \
  OPENAIAPIKEY=sk-xxx \

  HFTOKEN=hfxxx

import modal

app = modal.App("secrets-example")

@app.function(secrets=[modal.Secret.fromname("my-api-keys")])
def callopenai():

    import os
    import openai

    # Secrets are available as environment variables
    client = openai.OpenAI(apikey=os.environ["OPENAIAPIKEY"])

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Hello!"}]
    )
    return response.choices[0].message.content

Secrets from local environment variables
@app.function(secrets=[modal.Secret.fromdotenv()])

def useenvsecrets():

    import os
    token = os.environ.get("HFTOKEN")
    return f"Token loaded: {token[:10]}..."

import modal

app = modal.App("parallel-processing")

image = modal.Image.debianslim().pipinstall("Pillow", "torch", "torchvision")

@app.function(gpu="T4", image=image)
def processimage(imageurl: str) -> dict:
    """Process a single image - runs on GPU."""
    # Simulate image processing
    import time
    time.sleep(2)  # Simulate processing time
    return {"url": imageurl, "status": "processed"}


@app.localentrypoint()
def main():
    # Process 100 images in parallel
    imageurls = [f"https://example.com/image{i}.jpg" for i in range(100)]

    # .map() runs the function in parallel across multiple containers
    results = list(processimage.map(imageurls))

    print(f"Processed {len(results)} images")
    for r in results[:5]:
        print(r)

@app.function(gpu="T4")
def trainmodel(learningrate: float, batchsize: int, epochs: int) -> dict:

    """Train a model with specific hyperparameters."""
    # Simulate training
    accuracy = 0.85 + learningrate  10 - batchsize  0.001

    return {
        "lr": learningrate,
        "batchsize": batchsize,
        "epochs": epochs,
        "accuracy": round(accuracy, 4)
    }

@app.localentrypoint()

def hyperparametersearch():
    # Parallel grid search
    configs = [
        (0.001, 16, 10),
        (0.001, 32, 10),
        (0.01, 16, 10),
        (0.01, 32, 10),
        (0.1, 16, 10),
        (0.1, 32, 10),
    ]

    results = list(trainmodel.starmap(configs))


    # Find the best configuration
    best = max(results, key=lambda x: x["accuracy"])
    print(f"Best config: {best}")

import modal

app = modal.App("model-serving")

Define image with all dependencies
image = modal.Image.debianslim(pythonversion="3.11").pipinstall(
    "torch",
    "transformers",
    "accelerate"
)

@app.cls(gpu="T4", image=image, containeridletimeout=300)
class SentimentAnalyzer:
    @modal.enter()
    def loadmodel(self):

        """Load model when container starts (only once)."""
        from transformers import pipeline
        self.classifier = pipeline(
            "sentiment-analysis",
            model="distilbert-base-uncased-finetuned-sst-2-english",
            device=0  # Use GPU
        )
        print("Model loaded successfully!")

    @modal.method()
    def predict(self, text: str) -> dict:
        """Predict sentiment of text."""
        result = self.classifier(text)[0]
        return {
            "text": text,
            "label": result["label"],
            "score": round(result["score"], 4)
        }

    @modal.method()
    def predictbatch(self, texts: list) -> list:
        """Predict sentiment for a batch of texts."""
        results = self.classifier(texts)
        return [
            {
                "text": text,
                "label": r["label"],
                "score": round(r["score"], 4)
            }
            for text, r in zip(texts, results)
        ]

@app.function()
@modal.webendpoint(method="POST")

def analyze(data: dict):
    analyzer = SentimentAnalyzer()
    text = data.get("text", "")

    if isinstance(text, list):
        return {"results": analyzer.predictbatch.remote(text)}
    else:
        return {"result": analyzer.predict.remote(text)}

import modal
from typing import Optional

Configuration
MODELNAME = "mistralai/Mistral-7B-Instruct-v0.2"

GPUCONFIG = modal.gpu.A100(count=1, size="40GB")

Image with pre-downloaded model
def downloadmodel():

    from transformers import AutoModelForCausalLM, AutoTokenizer
    import torch

    print(f"Downloading {MODELNAME}...")
    tokenizer = AutoTokenizer.frompretrained(MODELNAME)
    model = AutoModelForCausalLM.frompretrained(

        MODELNAME,
        torchdtype=torch.float16,

        devicemap="auto"
    )
    print("Model downloaded successfully!")

image = (
    modal.Image.debianslim(pythonversion="3.11")
    .pipinstall(

        "torch",
        "transformers",
        "accelerate",
        "scipy",
        "sentencepiece",
        "protobuf"
    )
    .runfunction(
        downloadmodel,

        secrets=[modal.Secret.fromname("huggingface-token")],
        gpu=GPUCONFIG

    )
)

app = modal.App("hf-model-api", image=image)

@app.cls(
    gpu=GPUCONFIG,
    containeridletimeout=300,
    allowconcurrentinputs=10,
    secrets=[modal.Secret.fromname("huggingface-token")]

)
class LLMInference:
    @modal.enter()
    def load(self):
        """Load model to GPU when container starts."""
        from transformers import AutoModelForCausalLM, AutoTokenizer
        import torch

        self.tokenizer = AutoTokenizer.frompretrained(MODELNAME)

        self.model = AutoModelForCausalLM.frompretrained(
            MODELNAME,

            torchdtype=torch.float16,
            devicemap="auto"

        )

        # Set pad token
        if self.tokenizer.padtoken is None:
            self.tokenizer.padtoken = self.tokenizer.eostoken

        print("Model loaded and ready!")

    @modal.method()
    def generate(
        self,
        prompt: str,
        maxnewtokens: int = 512,
        temperature: float = 0.7,
        topp: float = 0.9

    ) -> str:
        """Generate text from a prompt."""
        import torch

        # Format prompt for instruction model
        messages = [{"role": "user", "content": prompt}]
        formatted = self.tokenizer.applychattemplate(

            messages,
            tokenize=False,
            addgenerationprompt=True

        )

        inputs = self.tokenizer(formatted, returntensors="pt").to("cuda")

        with torch.nograd():

            outputs = self.model.generate(
                inputs,

                maxnewtokens=maxnewtokens,

                temperature=temperature,
                topp=topp,

                dosample=True,
                padtokenid=self.tokenizer.padtokenid
            )

        # Decode only generated tokens (not the prompt)
        newtokens = outputs[0][inputs["inputids"].shape[1]:]
        response = self.tokenizer.decode(newtokens, skipspecialtokens=True)


        return response.strip()

    @modal.method()
    def healthcheck(self) -> dict:
        """Check if the model is loaded."""
        import torch
        return {
            "status": "healthy",
            "model": MODELNAME,

            "gpu": torch.cuda.getdevicename(0),

            "gpumemoryallocated": f"{torch.cuda.memoryallocated() / 1e9:.2f} GB"
        }

Web endpoints
@app.function()
@modal.webendpoint(method="POST")

def generatetext(data: dict):
    """API endpoint for text generation."""
    llm = LLMInference()

    prompt = data.get("prompt", "")
    maxtokens = data.get("maxtokens", 512)
    temperature = data.get("temperature", 0.7)

    if not prompt:
        return {"error": "prompt is required"}, 400

    response = llm.generate.remote(
        prompt=prompt,
        maxnewtokens=maxtokens,

        temperature=temperature
    )

    return {
        "model": MODELNAME,
        "prompt": prompt,
        "response": response,
        "parameters": {
            "maxtokens": maxtokens,
            "temperature": temperature
        }
    }

@app.function()
@modal.webendpoint(method="GET")

def health():
    """Health check endpoint."""
    llm = LLMInference()
    return llm.healthcheck.remote()

Local testing
@app.localentrypoint()

def main():
    llm = LLMInference()

    # Test health check
    status = llm.healthcheck.remote()
    print(f"Status: {status}")

    # Test generation
    prompts = [
        "Explain machine learning in simple terms.",
        "Write a Python function to calculate fibonacci numbers.",
        "What are the benefits of serverless computing?"
    ]

    for prompt in prompts:
        print(f"\nPrompt: {prompt}")
        response = llm.generate.remote(prompt)
        print(f"Response: {response[:200]}...")

# Test locally (runs in Modal cloud)
modal run hfmodelapi.py

Deploy as a service
modal deploy hfmodelapi.py

https://your-workspace--hf-model-api-generate-text.modal.run

# Health check
curl https://your-workspace--hf-model-api-health.modal.run

Generate text
curl -X POST https://your-workspace--hf-model-api-generate-text.modal.run \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain quantum computing", "maxtokens": 256}'

# Pre-download models in image build, not at runtime
image = (
    modal.Image.debianslim()
    .pipinstall("transformers", "torch")

    .runfunction(downloadmodel)  # Model is already in the image

)

Use containeridletimeout to keep containers warm

@app.function(containeridletimeout=600)  # 10 minutes idle before shutdown

def myfunction():
    pass

# Use timeout to prevent unexpected costs
@app.function(gpu="A100", timeout=3600)  # Maximum 1 hour
def longtraining():

    pass

Use T4 for inference (cheaper)
@app.function(gpu="T4")
def inference():
    pass

Use A100/H100 only for training
@app.function(gpu="A100")
def training():
    pass

# Run in detached mode
modal run --detach myapp.py

View logs
modal app logs my-app

Interactive shell in the container
modal shell myapp.py

# Allow multiple requests to a single container
@app.cls(
    gpu="T4",
    allowconcurrentinputs=20,  # Up to 20 concurrent requests

    containeridletimeout=300

)
class MyModel:
    @modal.enter()
    def setup(self):
        self.model = loadmodel()

    @modal.method()
    def predict(self, data):
        return self.model(data)

• Zero infrastructure management: No need to set up servers, Docker, or Kubernetes
• Pay-per-second billing: Pay only when your code runs
• Easy GPU access: From T4 to H100, just pick one
• Auto-scaling: From 0 to hundreds of containers automatically
• Developer experience*: Intuitive and modern Python API

Start using Modal for your next ML project and experience the difference. Focus on building models, not managing infrastructure.