Complete Llama Stack Tutorial: Meta's Official Framework for Building AI Applications with Llama Models

Llama Stack is an open-source framework from Meta that provides standardized APIs for building AI applications using Llama models. This framework unifies essential components such as inference, safety, memory, and agentic workflows into a single integrated ecosystem. In this tutorial, we will learn how to install, configure, and use Llama Stack to build production-ready AI applications.

What Is Llama Stack?

Llama Stack is an API specification and reference implementation developed by Meta to standardize how developers interact with Llama models. Instead of using Llama models directly through various different libraries, Llama Stack provides a single consistent interface for all AI development needs.

The main components of Llama Stack include:

• Inference API: Run Llama models for text generation, chat completion, and structured output
• Safety API: Apply content filtering and guardrails on model input/output
• Memory API: Store and retrieve conversation context and documents for RAG
• Agents API: Build AI agents that can use tools and complete complex tasks
• Evaluation API: Evaluate model performance on various benchmarks

The key advantage of Llama Stack is its portability. Code you write can run across different providers without significant changes, whether locally using Ollama, in the cloud using AWS Bedrock or Together AI, or on on-premise infrastructure.

Installation and Setup

Prerequisites

Make sure you have Python 3.10 or newer installed on your system.

python --version
Python 3.10+ required

Installing Llama Stack

Install the main Llama Stack package along with the client library:

pip install llama-stack llama-stack-client

For specific providers, install additional dependencies as needed:

# For Ollama (local inference)
pip install llama-stack[ollama]

For Together AI (cloud inference)
pip install llama-stack[together]

For Meta Reference (Meta's reference implementation)
pip install llama-stack[meta-reference]

Configuring the Llama Stack Server

Llama Stack works with a client-server architecture. First, we need to run a server that will serve API requests.

The quickest way to get started is using built-in templates:

# List available templates
llama stack list-templates

Use the Ollama template (for local development)
llama stack build --template ollama --image-type venv

Run the server
llama stack run ollama --port 8321

If you prefer Docker, the process is simpler:

# Run Llama Stack with Ollama via Docker
docker run -p 8321:8321 \
  -v ~/.llama:/root/.llama \
  llamastack/distribution-ollama \
  --port 8321

Make sure Ollama is installed and Llama models are downloaded:

# Install Ollama (if not already installed)
curl -fsSL https://ollama.ai/install.sh | sh

Download Llama models
ollama pull llama3.2:3b
ollama pull llama3.1:8b

Verifying Installation

After the server is running, verify by accessing the health check endpoint:

curl http://localhost:8321/v1/health
Response: {"status": "ok"}

List available models
curl http://localhost:8321/v1/models

Basic Usage

Initializing the Client

Create a new Python file and initialize the Llama Stack client:

from llamastackclient import LlamaStackClient

Connect to Llama Stack server
client = LlamaStackClient(baseurl="http://localhost:8321")


Verify connection by listing available models
models = client.models.list()
for model in models:
    print(f"Model: {model.identifier}, Provider: {model.providerid}")

Basic Chat Completion

Here's how to perform a simple chat completion:

from llamastackclient import LlamaStackClient

client = LlamaStackClient(baseurl="http://localhost:8321")


response = client.inference.chatcompletion(
    modelid="llama3.1-8b",

    messages=[
        {"role": "system", "content": "You are a helpful and friendly AI assistant."},
        {"role": "user", "content": "Explain what machine learning is in 3 sentences."}
    ],
    temperature=0.7,
    maxtokens=512
)

print(response.completionmessage.content)

from llamastackclient import LlamaStackClient

client = LlamaStackClient(baseurl="http://localhost:8321")

stream = client.inference.chatcompletion(

    modelid="llama3.1-8b",
    messages=[
        {"role": "user", "content": "Write a short poem about AI technology."}
    ],
    stream=True
)

for chunk in stream:
    if hasattr(chunk, 'event') and chunk.event.delta:
        print(chunk.event.delta, end="", flush=True)

print()  # newline after streaming completes

Structured Output

Llama Stack supports structured output using JSON schema:

from llamastackclient import LlamaStackClient
import json

client = LlamaStackClient(baseurl="http://localhost:8321")


response = client.inference.chatcompletion(
    modelid="llama3.1-8b",

    messages=[
        {
            "role": "user",
            "content": "Provide information about 3 popular programming languages for AI."
        }
    ],
    responseformat={
        "type": "jsonschema",

        "jsonschema": {
            "name": "programminglanguages",

            "schema": {
                "type": "object",
                "properties": {
                    "languages": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "properties": {
                                "name": {"type": "string"},
                                "usecase": {"type": "string"},
                                "popularityrank": {"type": "integer"}

                            },
                            "required": ["name", "usecase", "popularityrank"]

                        }
                    }
                },
                "required": ["languages"]
            }
        }
    }
)

result = json.loads(response.completionmessage.content)
print(json.dumps(result, indent=2))

Advanced Usage

Building RAG with Memory API

One of the most powerful features of Llama Stack is the Memory API that enables Retrieval-Augmented Generation (RAG). Here's an implementation example:

from llamastackclient import LlamaStackClient

client = LlamaStackClient(baseurl="http://localhost:8321")


Create a memory bank to store documents
bankid = "knowledge-base-01"
client.memorybanks.register(

    memorybankid=bankid,
    params={
        "memorybanktype": "vector",
        "embeddingmodel": "all-MiniLM-L6-v2",

        "chunksizeintokens": 512,
        "overlapsizeintokens": 64

    }
)

Insert documents into the memory bank
documents = [
    {
        "documentid": "doc-1",
        "content": """
        Llama 3.1 is a large language model from Meta available in 8B, 70B,
        and 405B parameter sizes. The model supports context windows up to
        128K tokens and has been trained on multilingual data.
        """,
        "metadata": {"source": "meta-docs", "topic": "llama-models"}
    },
    {
        "documentid": "doc-2",

        "content": """
        Llama Stack provides standardized APIs for inference, safety, memory,
        and agents. The framework supports various providers including Ollama,
        Together AI, AWS Bedrock, and Fireworks AI.
        """,
        "metadata": {"source": "meta-docs", "topic": "llama-stack"}
    }
]

client.memory.insert(
    bankid=bankid,

    documents=documents
)

Query with RAG
query = "What parameter sizes are available for Llama 3.1?"

Search for relevant documents
results = client.memory.query(
    bankid=bankid,

    query=query,
    params={"maxchunks": 3}
)

Build context from search results
context = "\n".join([chunk.content for chunk in results.chunks])

Generate answer with context
response = client.inference.chatcompletion(

    modelid="llama3.1-8b",
    messages=[
        {
            "role": "system",
            "content": f"Answer the question based on the following context:\n\n{context}"
        },
        {"role": "user", "content": query}
    ]
)

print(response.completionmessage.content)

from llamastackclient import LlamaStackClient
from llamastackclient.types.agentcreateparams import AgentConfig


client = LlamaStackClient(baseurl="http://localhost:8321")

Define a custom tool
searchtool = {

    "type": "function",
    "function": {
        "name": "searchdatabase",
        "description": "Search for product information in the database",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": "Search keywords"
                },
                "category": {
                    "type": "string",
                    "enum": ["electronics", "clothing", "food"],
                    "description": "Product category"
                }
            },
            "required": ["query"]
        }
    }
}

Create agent configuration
agentconfig = AgentConfig(

    model="llama3.1-8b",
    instructions="""You are a shopping assistant that helps customers find products.
    Use the searchdatabase tool to look up relevant product information.
    Provide clear and informative recommendations.""",
    tools=[searchtool],

    enablesessionpersistence=True,

    maxinferiters=5

)

Create the agent
agent = client.agents.create(agentconfig=agentconfig)

agentid = agent.agentid


Create a new session
session = client.agents.session.create(
    agentid=agentid,

    sessionname="shopping-session-01"
)
sessionid = session.sessionid

Send a message to the agent
response = client.agents.turn.create(
    agentid=agentid,
    sessionid=sessionid,
    messages=[
        {"role": "user", "content": "Find me a gaming laptop under $1000"}
    ],
    stream=False
)

Process response and tool calls
for step in response.steps:
    if step.steptype == "inference":

        print(f"Agent: {step.completionmessage.content}")
    elif step.steptype == "toolexecution":
        print(f"Tool called: {step.toolcalls}")

from llamastackclient import LlamaStackClient

client = LlamaStackClient(baseurl="http://localhost:8321")

Register a safety shield
client.shields.register(
    shieldid="content-filter",

    providershieldid="llama-guard"

)

Run safety check on input
inputcheck = client.safety.runshield(

    shieldid="content-filter",
    messages=[
        {"role": "user", "content": "How do I build a web application with Python?"}
    ]
)

if inputcheck.violation:

    print(f"Input rejected: {inputcheck.violation.usermessage}")

else:
    print("Input is safe, proceeding with inference...")

    response = client.inference.chatcompletion(
        modelid="llama3.1-8b",

        messages=[
            {"role": "user", "content": "How do I build a web application with Python?"}
        ]
    )

    # Safety check on output
    outputcheck = client.safety.runshield(

        shieldid="content-filter",
        messages=[
            {"role": "assistant", "content": response.completionmessage.content}

        ]
    )

    if outputcheck.violation:
        print("Output filtered due to safety policy violation.")
    else:
        print(response.completionmessage.content)

# config.yaml
apis:
inference
safety
memory
agents

providers:
  inference:
providerid: ollama-local
      providertype: remote::ollama

      config:
        url: http://localhost:11434
providerid: together-cloud
      providertype: remote::together

      config:
        apikey: ${TOGETHERAPIKEY}

  safety:
providerid: llama-guard

      providertype: inline::llama-guard

      config:
        model: llama-guard-3-1b

  memory:
providerid: faiss-local

      providertype: inline::faiss

      config:
        kvstore:
          type: sqlite
          dbpath: ~/.llama/memory.db


  agents:
providerid: meta-reference
      providertype: inline::meta-reference

      config: {}

models:
metadata: {}
    modelid: llama3.2-3b

    providerid: ollama-local

    providermodelid: llama3.2:3b

metadata: {}
    modelid: llama3.1-70b

    providerid: together-cloud

    providermodelid: meta-llama/Llama-3.1-70B-Instruct-Turbo

llama stack run config.yaml --port 8321

from llamastackclient import LlamaStackClient

client = LlamaStackClient(baseurl="http://localhost:8321")

Register a benchmark dataset
client.datasets.register(
    datasetid="qa-benchmark",

    datasetschema={
        "type": "map",
        "key": {"type": "string"},
        "value": {
            "type": "object",
            "properties": {
                "question": {"type": "string"},
                "expectedanswer": {"type": "string"}

            }
        }
    },
    url="file:///path/to/benchmark.jsonl"
)

Run evaluation
evaljob = client.eval.runeval(

    taskid="qa-evaluation",
    taskconfig={

        "type": "benchmark",
        "evalcandidate": {
            "type": "model",
            "model": "llama3.1-8b",
            "samplingparams": {

                "temperature": 0.1,
                "maxtokens": 256
            }
        },
        "scoringfunctions": ["accuracy", "rougel", "bleu"]
    }
)

Get evaluation results
results = client.eval.jobresult(

    taskid="qa-evaluation",
    jobid=evaljob.jobid

)

for metric, score in results.scores.items():
    print(f"{metric}: {score:.4f}")

# docker-compose.yml
version: '3.8'

services:
  llama-stack:
    image: llamastack/distribution-ollama
    ports:
"8321:8321"
    volumes:

llama-data:/root/.llama
./config.yaml:/app/config.yaml
    environment:

TOGETHERAPIKEY=${TOGETHERAPIKEY}

    command: ["--config", "/app/config.yaml", "--port", "8321"]

    dependson:
ollama
    restart: unless-stopped


  ollama:
    image: ollama/ollama
    ports:
"11434:11434"
    volumes:

ollama-data:/root/.ollama
    deploy:

      resources:
        reservations:
          devices:
driver: nvidia
              count: 1

              capabilities: [gpu]
    restart: unless-stopped

volumes:
  llama-data:
  ollama-data:

Monitoring and Logging

For production, add telemetry configuration:

from llamastackclient import LlamaStackClient

client = LlamaStackClient(baseurl="http://localhost:8321")


Enable telemetry
client.telemetry.logevent(
    event={
        "type": "structuredlog",

        "payload": {
            "message": "Inference request completed",
            "model": "llama3.1-8b",
            "latencyms": 1250,
            "tokensgenerated": 128

        }
    }
)

Query telemetry data
spans = client.telemetry.queryspans(
    attributefilters=[

        {"key": "model", "value": "llama3.1-8b"}
    ],
    limit=100
)

for span in spans:
    print(f"Span: {span.spanid}, Duration: {span.durationms}ms")

response = client.inference.chatcompletion(
    modelid="llama3.1-8b",

    messages=messages,
    # Enable caching to reduce latency for repeated prompts
    samplingparams={
        "temperature": 0.7,
        "maxtokens": 1024,

    }
)

from llamastackclient import LlamaStackClient

client = LlamaStackClient(baseurl="http://localhost:8321")

try:
    response = client.inference.chatcompletion(

        modelid="llama3.1-8b",
        messages=[{"role": "user", "content": "Hello"}]
    )
    print(response.completion_message.content)
except ConnectionError:
    print("Cannot connect to Llama Stack server.")
except Exception as e:
    print(f"Error: {e}")

4. Use Safety Shields in Production

Always enable safety shields in production environments to prevent unwanted outputs. Configure Llama Guard as the default shield in your server configuration.

5. Optimize Memory Banks

For efficient RAG:

• Set chunk size according to document type (512 tokens for articles, 256 for FAQs)
• Use sufficient overlap (10-15% of chunk size) to maintain context
• Periodically clean memory banks to remove outdated documents

6. Monitor Resource Usage

Monitor GPU, RAM, and CPU usage when running models locally. The 8B model requires at least 8GB VRAM, while the 70B model needs at least 40GB VRAM, or you can use quantization to reduce memory requirements.

Comparison with Other Frameworks

| Aspect | Llama Stack | LangChain | LlamaIndex |

|--------|------------|-----------|------------|

| Focus | End-to-end Llama ecosystem | General LLM orchestration | RAG and data indexing |

| Model Support | Llama-first, extensible | Multi-model | Multi-model |

| Safety | Built-in Llama Guard | Requires integration | Requires integration |

| Agents | Native support | Comprehensive | Basic |

| Memory | Built-in vector store | Requires integration | Built-in |

| Deployment | Server-based | Library | Library |

Conclusion

Llama Stack is a powerful and well-structured framework for building AI applications using Llama models. With its consistent APIs, multi-provider support, and built-in features for safety, memory, and agents, Llama Stack simplifies AI application development from prototype to production.

Key takeaways:

Start with a local installation using Ollama, explore the available features, and gradually scale to production deployment as your understanding grows.