Truss: How to Package, Serve, and Deploy ML Models Cleanly

Hey folks, it's me again, Ruby. This time I want to talk about a tool that I feel a lot of ML practitioners overlook, even though it becomes incredibly useful once we hit the "the model is done, now how do I actually serve it to the outside world?" stage. That tool is called Truss, built by the team at Baseten.

If you have ever felt the pain of building a machine learning model that runs perfectly in a notebook, but the moment you try to deploy it as an API everything falls apart because of library version mismatches, missing system dependencies, wrong GPU configuration, or a Dockerfile that makes your head spin, then Truss is the answer to most of those problems. Truss gives us one standard way to package a model together with all its requirements, then serve it locally or in the cloud without having to become a DevOps expert first.

In this tutorial I'll walk you through everything from scratch: installing, creating your first Truss structure, understanding the generated files, trying local serving, adding Python and system dependencies, configuring GPUs, and finally deploying to Baseten. I'll also use concrete examples with HuggingFace models so it clicks. Let's dive in.

Introduction

What Truss actually is

Truss is an open-source framework for packaging, serving, and deploying ML models. The core idea is simple but powerful: every model is wrapped in a standard folder structure called a "Truss". Inside it there are two main things you need to fill in, namely the code to load and predict with the model (model/model.py), plus the environment configuration (config.yaml) that describes dependencies, resources, and other settings.

What makes Truss pleasant to use is that it handles the parts that are usually frustrating. You don't need to write a Dockerfile by hand, you don't need to set up a FastAPI/Flask server yourself, and you don't need to think about how a request comes in and gets routed to your predict function. All of that is handled by Truss through clear conventions.

Why I like this pattern

In my opinion Truss's biggest strength is the clean separation between model code and environment configuration. When we're coding, we just focus on model.py: how the model gets loaded into memory once at the start, and how each request is processed. Meanwhile the concerns of Python version, packages, system dependencies, and how much CPU/GPU is needed are all written declaratively in config.yaml. This separation makes our model far easier to reproduce on another machine, and far easier for a teammate to review.

On top of that, Truss is portable. The same Truss can run locally for testing, can be built into a Docker image, and can be deployed to Baseten for production. So there's no more "it works locally but not on the server" moment that usually keeps us up at night.

When you should use Truss

Truss is best for folks who have a model, whether it's a HuggingFace model, a custom PyTorch/TensorFlow model, or even a combined pipeline, and want to serve it as an inference service callable over HTTP. If your need is just running a script once, Truss might be overkill. But the moment you need a model that stays on standby, ready to accept requests any time, with clearly defined resource configuration, that's where Truss shines.

Installation

Prerequisites

Before starting, make sure you have Python 3.8 or newer. I recommend creating a virtual environment first to keep things clean and avoid clashing with other project dependencies. If you want to serve via Docker (and this is the closest thing to production conditions), make sure Docker is also installed and running on your machine.

First we create a virtual environment and activate it:

python3 -m venv venv
source venv/bin/activate

If you're on Windows, activation is slightly different:

python -m venv venv
venv\Scripts\activate

Install Truss

Installing Truss is super easy, just one pip command:

pip install truss

Once it's done, we can check the version to confirm the install succeeded:

truss version

If a version number appears, Truss is ready to go. Truss provides a command-line tool called truss that we'll use often for initialization, local serving, and deployment.

Log in to Baseten (optional for now)

If you already plan to deploy to Baseten, you can prepare an API key from the Baseten dashboard. But this isn't mandatory at the early stage, because local serving doesn't require an account at all. We'll cover this in more detail in the deployment section. For now just make sure Truss is installed and the truss command works.

Basic Usage

Initializing your first Truss

Now let's create our first Truss. The command is truss init followed by the folder name we want to create. For example, say I want to build a sentiment text classification model:

truss init sentiment-classifier

When run, Truss will generate a standard folder structure for us. In some versions it asks a few interactive questions, but essentially it scaffolds something ready to be filled in. Once done, go into the folder and look inside:

cd sentiment-classifier
ls -la

Understanding the generated structure

The basic structure of a Truss looks roughly like this:

sentiment-classifier/
├── config.yaml
├── model/
│   └── model.py
├── data/
└── packages/

Let me explain each one so you understand its purpose:

The model/ folder contains model.py, and this is the heart of it. This is where we write the load and predict logic. The config.yaml file is where we declare all environment configuration: Python dependencies, system dependencies, resources (CPU/GPU/memory), model name, and more. The data/ folder is used to place model weight files or other assets that need to be bundled along. The packages/ folder is for additional Python code we want to import in model.py, such as custom utility modules.

The contents of model.py

The default model/model.py is usually a Model class with two important methods: load() and predict(). The concept is really important to understand:

The load() method is called once when the server first starts. This is where we load the model into memory. Since it's only called once, heavy operations like downloading weights and initializing the model go here so they aren't repeated on every request.

The predict() method is called every time a request comes in. Its job is to receive input (usually a dictionary parsed from the request JSON), then return output that gets sent back as the response.

Now let's fill in model/model.py with a real example using a sentiment model from HuggingFace:

from transformers import pipeline


class Model:
    def init(self, kwargs):

        # kwargs contains config, datadir, secrets, etc. from Truss

        self.model = None

    def load(self):
        # Called once at server start. Load the model into memory here.
        self.model = pipeline(

            "sentiment-analysis",
            model="distilbert-base-uncased-finetuned-sst-2-english",
        )

    def predict(self, modelinput):
        # Called on every request. modelinput is a dict parsed from JSON.

        text = modelinput.get("text", "")
        result = self.model(text)

        return {"predictions": result}

modelname: sentiment-classifier
pythonversion: py311

requirements:
torch==2.3.0
transformers==4.44.0
resources:

  cpu: "1"
  memory: 2Gi
  usegpu: false

truss predict --target-directory . -d '{"text": "I really love this tool"}'

truss image build .
truss run-image .

curl -X POST http://localhost:8080/v1/models/model:predict \
  -H "Content-Type: application/json" \
  -d '{"text": "Deploying a model became so easy"}'

requirements:
torch==2.3.0
transformers==4.44.0
pillow==10.4.0
numpy==1.26.4

systempackages:
ffmpeg
libgl1-mesa-glx
libglib2.0-0

resources:
  cpu: "4"
  memory: 16Gi
  usegpu: true

  accelerator: A10G

import base64
import io

from PIL import Image
from transformers import pipeline


class Model:
    def init(self, kwargs):
        self.model = None


    def load(self):
        self.model = pipeline(
            "image-classification",
            model="google/vit-base-patch16-224",
        )

    def predict(self, modelinput):

        # Receive the image as a base64 string
        imageb64 = modelinput.get("imagebase64", "")
        imagebytes = base64.b64decode(imageb64)
        image = Image.open(io.BytesIO(imagebytes)).convert("RGB")


        results = self.model(image)
        # Take the top-3 predictions
        top = results[:3]
        return {"predictions": top}

Then we adjust config.yaml for an image model with GPU:

modelname: vit-image-classifier
pythonversion: py311
requirements:
torch==2.3.0
transformers==4.44.0
pillow==10.4.0
systempackages:

libgl1-mesa-glx
resources:

  cpu: "2"
  memory: 8Gi
  usegpu: true
  accelerator: T4

With this setup, you have an image classification model ready to run on a GPU. For local testing, you can encode an image to base64 first via a small Python snippet, then send it to truss predict.

Loading a model from a local file (bundled weights)

Sometimes we don't want to download the model from the internet every time we deploy, either because the model is custom trained by ourselves, or to make startup faster. We can place weight files in the data/ folder and load from there. Truss will bundle the contents of the data/ folder into the image, and provides its path via the datadir argument:

import os
import joblib


class Model:
    def init(self, kwargs):

        self.datadir = kwargs["datadir"]
        self.model = None


    def load(self):
        modelpath = os.path.join(self.datadir, "model.joblib")
        self.model = joblib.load(modelpath)

    def predict(self, modelinput):

        features = modelinput.get("features", [])
        prediction = self.model.predict([features])

        return {"prediction": prediction.tolist()}

secrets:
  hfaccesstoken: null

class Model:
    def init(self, kwargs):
        self.secrets = kwargs["secrets"]
        self.model = None


    def load(self):
        token = self.secrets["hfaccesstoken"]
        # use the token to load a private model
        self.model = pipeline(

            "text-generation",
            model="my-org/private-model",
            token=token,
        )

    def predict(self, modelinput):
        return {"output": self.model(modelinput.get("prompt", ""))}

This way your sensitive token is never stored in the code or in git. This is really important for security.

Deploying to Baseten

Now the climax: deploying to Baseten so our model can be accessed from anywhere via a production endpoint. First, you need to log in using an API key from the Baseten dashboard:

truss login

This command will ask for your API key. Once login succeeds, deploying is just one command from inside the Truss folder:

truss push

Truss will package the entire model, build the image on Baseten's infrastructure, then deploy it as a served model. The process shows progress and at the end tells you your model's endpoint. Once it's live, we can call it over HTTP by including the Baseten API key in the header:

curl -X POST https://model-xxxxx.api.baseten.co/development/predict \
  -H "Authorization: Api-Key YOURBASETENAPI_KEY" \
  -d '{"text": "The model is live in production"}'

Baseten also has the concept of development and production environments. During development, each truss push immediately updates the model for fast iteration. Once you're confident, you can promote to a production environment that's more stable and doesn't change on every push.

Publishing to production

If you want to deploy straight to the production environment, you can add a flag:

truss push --publish

With the --publish flag, Truss creates a deployment tagged as production, not just a development one that keeps changing. This is what you use when the model is ready for real users.

Best Practices

After spending a decent amount of time with Truss, there are a few habits that I think will make your life much calmer. Let me sum them up here.

Always pin dependency versions

I already touched on this, but it's so important I'll repeat it. Always pin package versions in requirements using ==. A reproducible model is gold. If versions aren't locked, you'll hit bugs that appear randomly just because a library updated behind the scenes. Lock all versions, including torch, transformers, and other critical libraries.

Test locally before pushing

Don't just truss push without trying locally first. Always run truss predict or serve via Docker locally to make sure load() and predict() work as they should. Deploying to the cloud costs time and resources, so catching an error locally is far cheaper and faster than waiting for a cloud build to fail only to realize there was a typo.

Put heavy operations in load(), not predict()

This is a mistake I often see: people put model loading or heavy setup inside predict(). As a result every request gets really slow because the model keeps being reloaded. Remember, load() is called only once, predict() is called on every request. Everything heavy that only needs to happen once, put it in load().

Match resources to actual needs

Don't just request an A100 GPU when your small model actually runs fine on CPU. Resources cost money, especially GPUs. Start small, measure performance and latency, then scale up only if truly needed. Conversely, don't be so stingy that the model runs out of memory and crashes. Measure the real needs through testing.

Clean up preprocessing and validate input

Since predict() receives raw input from the user, always validate and provide safe defaults. If the text field is missing, don't let the server crash. Handle it neatly so our API is resilient to weird input. This makes your service far more professional and stable.

Leverage the packages folder for reusable code

If your model.py starts getting bloated, move logic that can be separated into the packages/ folder. For example preprocessing functions, postprocessing, or other utilities. This keeps the code tidy and easy to test in isolation.

Manage secrets properly

Once again, never hardcode tokens or credentials in your code. Use Truss's secrets mechanism. Besides being more secure, this also makes your model easy to move between environments without editing code.

Conclusion

Alright folks, we've traveled quite far with Truss from Baseten. We started with installing via pip install truss, creating the first structure with truss init, understanding the roles of model/model.py with its load() and predict() methods, plus config.yaml for managing dependencies and resources. Then we tried serving locally via truss predict and Docker, adding Python and system dependencies, configuring GPUs for heavy models, and finally deploying to Baseten with truss push.

What I like about Truss is that it gives a clear and consistent structure to something that's usually messy. The separation between model code and environment configuration makes our model reproducible and easy to maintain. And because the same Truss runs from local to production, we're no longer haunted by the "it only works locally" moment.

My advice is to just practice right away. Take one of your favorite models from HuggingFace, wrap it into a Truss, serve it locally, and if you're brave, deploy it to Baseten. Once you feel how smooth the flow is, I'm sure you'll get hooked. Happy experimenting, and see you in the next tutorial. Keep up the learning, folks.