llama.cpp and GGUF Quantization: Local LLM Deployment

Introduction

Running Large Language Models (LLMs) locally without depending on cloud APIs is a goal many developers and organizations share. With llama.cpp, this becomes a reality. llama.cpp is an LLM inference framework written in pure C/C++, designed to run large models efficiently on consumer hardware, including ordinary laptops.

The GGUF (GPT-Generated Unified Format) is a model format optimized for llama.cpp, supporting various quantization levels that allow you to trade off between model quality and memory usage according to your hardware capabilities.

In this tutorial, we will learn how to install llama.cpp, download GGUF models from HuggingFace, understand quantization levels, and build a fully functional local chatbot.

Prerequisites

• Computer with at least 8GB RAM (16GB+ recommended)
• At least 10GB free storage space
• Python 3.8 or later
• Git and CMake (for building from source)
• Optional: NVIDIA GPU with CUDA or Apple Silicon for acceleration

Installing llama.cpp

Method 1: Build from Source (Recommended)

# Clone repository
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

Basic build (CPU only)
make

Or using CMake
mkdir build && cd build
cmake ..
cmake --build . --config Release

Build with GPU Acceleration

# CUDA (NVIDIA GPU)
make GGMLCUDA=1


Or via CMake
mkdir build && cd build
cmake .. -DGGMLCUDA=ON
cmake --build . --config Release

Metal (Apple Silicon / macOS)
make GGMLMETAL=1


Or via CMake
mkdir build && cd build
cmake .. -DGGMLMETAL=ON
cmake --build . --config Release

Vulkan (cross-platform GPU)
make GGMLVULKAN=1

# Basic installation (CPU only)
pip install llama-cpp-python

With CUDA support
CMAKEARGS="-DGGMLCUDA=on" pip install llama-cpp-python


With Metal support (macOS)
CMAKEARGS="-DGGMLMETAL=on" pip install llama-cpp-python


With Vulkan support
CMAKEARGS="-DGGMLVULKAN=on" pip install llama-cpp-python


Force reinstall if upgrading
pip install llama-cpp-python --force-reinstall --no-cache-dir

# For source build
./llama-cli --version

For Python
python -c "from llamacpp import Llama; print('llama-cpp-python installed successfully')"

Understanding GGUF Format and Quantization Levels

What is GGUF?

GGUF is a file format specifically designed for storing quantized LLM models. It replaces the older GGML format and offers:

• Better backward and forward compatibility
• More complete metadata (tokenizer info, model parameters, etc.)
• Faster loading times
• Support for multiple model architectures

Quantization Levels

Quantization is the process of reducing the numerical precision of model parameters to save memory and increase speed, with a trade-off in output quality.

| Quantization | Bits | Model Size (7B) | RAM Needed | Quality | Use Case |

|-------------|------|-----------------|------------|---------|----------|

| F16 | 16 | ~14 GB | ~16 GB | Best | Reference, evaluation |

| Q80 | 8 | ~7.5 GB | ~10 GB | Excellent | Production (if RAM allows) |

| Q6K | 6 | ~5.5 GB | ~8 GB | Very Good | Balance quality/performance |

| Q5KM | 5 | ~5.0 GB | ~7.5 GB | Good | General recommendation |

| Q5KS | 5 | ~4.8 GB | ~7 GB | Good | Slightly smaller |

| Q4KM | 4 | ~4.0 GB | ~6.5 GB | Decent | Most popular |

| Q4KS | 4 | ~3.8 GB | ~6 GB | Fair | Limited RAM |

| Q3KM | 3 | ~3.3 GB | ~5.5 GB | Degraded | Only if RAM is very limited |

| Q2K | 2 | ~2.7 GB | ~5 GB | Low | Experimental only |

• Q4KM: Best balance between size and quality (most popular)
• Q5KM: For better quality with slightly more RAM
• Q80: If RAM is not a concern, quality approaches F16

GGUF Naming Convention

model-name-{size}-{type}.{quantization}.gguf

Examples:
Meta-Llama-3-8B-Instruct-Q4KM.gguf
mistral-7b-instruct-v0.3-Q5KM.gguf
phi-3-mini-4k-instruct-Q80.gguf

# Install huggingfacehub
pip install huggingfacehub


Download a specific model
huggingface-cli download \
    TheBloke/Llama-2-7B-Chat-GGUF \
    llama-2-7b-chat.Q4KM.gguf \

    --local-dir ./models

Download from bartowski (popular GGUF source)
huggingface-cli download \
    bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
    Meta-Llama-3.1-8B-Instruct-Q4KM.gguf \

    --local-dir ./models

from huggingfacehub import hfhubdownload

Download model
modelpath = hfhubdownload(

    repoid="bartowski/Meta-Llama-3.1-8B-Instruct-GGUF",
    filename="Meta-Llama-3.1-8B-Instruct-Q4KM.gguf",
    localdir="./models"

)

print(f"Model downloaded to: {modelpath}")

Recommended Popular Models

# Llama 3.1 8B (Meta, general purpose)
huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
    Meta-Llama-3.1-8B-Instruct-Q4KM.gguf --local-dir ./models

Mistral 7B (Mistral AI, efficient)
huggingface-cli download TheBloke/Mistral-7B-Instruct-v0.2-GGUF \
    mistral-7b-instruct-v0.2.Q4KM.gguf --local-dir ./models

Phi-3 Mini (Microsoft, compact)
huggingface-cli download bartowski/Phi-3.5-mini-instruct-GGUF \
    Phi-3.5-mini-instruct-Q4KM.gguf --local-dir ./models

Qwen2 7B (Alibaba, multilingual)
huggingface-cli download Qwen/Qwen2-7B-Instruct-GGUF \
    qwen2-7b-instruct-q4km.gguf --local-dir ./models

CLI Usage

Basic Inference

# Text completion
./llama-cli -m ./models/Meta-Llama-3.1-8B-Instruct-Q4KM.gguf \
    -p "Explain what machine learning is in 3 sentences:" \
    -n 256 \
    --temp 0.7

Interactive chat mode
./llama-cli -m ./models/Meta-Llama-3.1-8B-Instruct-Q4KM.gguf \
    --interactive \
    --color \
    -n -1 \
    --temp 0.7 \
    --top-p 0.9 \
    --repeat-penalty 1.1

Important CLI Parameters

./llama-cli -m model.gguf \
    -p "prompt text" \
    -n 512              # Max tokens to generate
    --temp 0.7          # Temperature (0.0 = deterministic, 1.0 = creative)
    --top-p 0.9         # Top-p sampling
    --top-k 40          # Top-k sampling
    --repeat-penalty 1.1 # Repetition penalty
    -c 4096             # Context length
    -t 8                # Number of CPU threads
    -ngl 35             # Number of layers on GPU (for GPU offloading)
    --seed 42           # Random seed for reproducibility

GPU Offloading

# Offload all layers to GPU
./llama-cli -m model.gguf -ngl 999 -p "Hello"

Offload partial layers (for GPUs with limited VRAM)
./llama-cli -m model.gguf -ngl 20 -p "Hello"

Check how many layers the model has
./llama-cli -m model.gguf --verbose-prompt -p "test" -n 1

Python Bindings: llama-cpp-python

Basic Usage

from llamacpp import Llama

Load model
llm = Llama(
    modelpath="./models/Meta-Llama-3.1-8B-Instruct-Q4KM.gguf",
    nctx=4096,        # Context window

    nthreads=8,       # CPU threads
    ngpulayers=35,   # GPU layers (0 for CPU only)
    verbose=False
)

Text completion
output = llm(
    "Explain what a neural network is:",
    maxtokens=256,

    temperature=0.7,
    topp=0.9,
    stop=["###", "\n\n\n"]
)

print(output["choices"][0]["text"])

Chat Completion (OpenAI-Compatible Format)

from llamacpp import Llama

llm = Llama(
    modelpath="./models/Meta-Llama-3.1-8B-Instruct-Q4KM.gguf",
    nctx=4096,

    ngpulayers=-1,  # -1 = all layers to GPU

    chatformat="llama-3"  # Chat format matching the model
)

Single turn
response = llm.createchatcompletion(
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant."},
        {"role": "user", "content": "What are the advantages of running LLMs locally?"}
    ],
    maxtokens=512,

    temperature=0.7
)

print(response["choices"][0]["message"]["content"])

from llamacpp import Llama

llm = Llama(
    modelpath="./models/Meta-Llama-3.1-8B-Instruct-Q4KM.gguf",

    nctx=4096,
    ngpulayers=-1
)

Streaming output
stream = llm.createchatcompletion(
    messages=[
        {"role": "system", "content": "You are an AI assistant."},
        {"role": "user", "content": "Write a short poem about programming"}
    ],
    maxtokens=256,

    stream=True
)

for chunk in stream:
    delta = chunk["choices"][0]["delta"]
    if "content" in delta:
        print(delta["content"], end="", flush=True)
print()  # Newline at the end

from llamacpp import Llama

llm = Llama(
    modelpath="./models/Meta-Llama-3.1-8B-Instruct-Q4KM.gguf",

    nctx=4096,
    ngpulayers=-1
)

Store conversation history
conversationhistory = [

    {"role": "system", "content": "You are an AI assistant expert in Python programming."}
]

def chat(usermessage):
    conversationhistory.append({"role": "user", "content": usermessage})

    response = llm.createchatcompletion(
        messages=conversationhistory,

        maxtokens=512,
        temperature=0.7
    )

    assistantmessage = response["choices"][0]["message"]["content"]

    conversationhistory.append({"role": "assistant", "content": assistantmessage})


    return assistantmessage

Multi-turn conversation
print(chat("What is list comprehension in Python?"))
print(chat("Give me 3 examples of its usage"))
print(chat("How does it perform compared to regular for loops?"))

OpenAI-Compatible Server

llama.cpp provides a server that is compatible with the OpenAI API, so you can use local models as a drop-in replacement for OpenAI.

Running the Server

# From source build
./llama-server \
    -m ./models/Meta-Llama-3.1-8B-Instruct-Q4KM.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    -c 4096 \
    -ngl 35 \
    --embedding

Or via Python
python -m llamacpp.server \

    --model ./models/Meta-Llama-3.1-8B-Instruct-Q4KM.gguf \

    --host 0.0.0.0 \
    --port 8080 \
    --nctx 4096 \
    --ngpulayers 35

Using with OpenAI Python SDK

from openai import OpenAI

Point to local server
client = OpenAI(
    baseurl="http://localhost:8080/v1",

    apikey="not-needed"  # API key not required for local server
)

Chat completion (exactly the same as OpenAI API)
response = client.chat.completions.create(
    model="local-model",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing in simple terms"}
    ],
    maxtokens=512,

    temperature=0.7
)

print(response.choices[0].message.content)

Streaming
stream = client.chat.completions.create(
    model="local-model",
    messages=[
        {"role": "user", "content": "Write a Python function to sort a list"}
    ],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

# Chat completion
curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Hello!"}
        ],
        "maxtokens": 256,
        "temperature": 0.7
    }'

Text embedding
curl http://localhost:8080/v1/embeddings \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Machine learning is fascinating",
        "model": "local-model"
    }'

Memory Requirements and Performance Benchmarks

Estimating RAM Requirements

Formula: RAM = (Model Size in GB) + (Context Length  0.5MB per 1K tokens) + 1GB overhead

Example for Llama 3.1 8B Q4KM:
Model size: ~4.5 GB
Context 4096 tokens: ~2 GB
Overhead: ~1 GB
Total: ~7.5 GB RAM

Example for Llama 3.1 70B Q4KM:
Model size: ~40 GB
Context 4096 tokens: ~4 GB
Overhead: ~2 GB
Total: ~46 GB RAM

import time
from llamacpp import Llama


llm = Llama(
    modelpath="./models/Meta-Llama-3.1-8B-Instruct-Q4KM.gguf",
    nctx=2048,

    ngpulayers=-1,

    verbose=True  # Will display performance statistics
)

Benchmark
prompt = "Explain the theory of relativity in detail:"
starttime = time.time()

output = llm(prompt, maxtokens=256)


elapsed = time.time() - starttime
tokensgenerated = output["usage"]["completiontokens"]
tokenspersecond = tokensgenerated / elapsed


print(f"Tokens generated: {tokensgenerated}")
print(f"Time elapsed: {elapsed:.2f}s")
print(f"Speed: {tokenspersecond:.1f} tokens/s")

import sys
from llamacpp import Llama


class LocalChatbot:
    def init(self, modelpath, systemprompt=None):

        print("Loading model... (this may take a moment)")
        self.llm = Llama(
            modelpath=modelpath,

            nctx=4096,
            ngpulayers=-1,
            verbose=False
        )

        self.systemprompt = systemprompt or (
            "You are an intelligent and helpful AI assistant. "
            "Respond in the same language as the user's question. "
            "Provide informative yet concise answers."
        )

        self.conversation = [
            {"role": "system", "content": self.systemprompt}

        ]
        print("Model loaded. Ready to chat!\n")

    def chat(self, userinput):
        self.conversation.append({"role": "user", "content": userinput})


        response = self.llm.createchatcompletion(

            messages=self.conversation,
            maxtokens=1024,
            temperature=0.7,
            topp=0.9,

            repeatpenalty=1.1,
            stream=True
        )

        fullresponse = ""

        for chunk in response:
            delta = chunk["choices"][0]["delta"]
            if "content" in delta:
                token = delta["content"]
                print(token, end="", flush=True)
                fullresponse += token

        print()  # Newline
        self.conversation.append({"role": "assistant", "content": fullresponse})


        # Trim conversation if too long
        if len(self.conversation) > 20:
            self.conversation = [self.conversation[0]] + self.conversation[-10:]

        return fullresponse

    def reset(self):
        self.conversation = [
            {"role": "system", "content": self.systemprompt}

        ]
        print("Conversation reset.\n")

    def run(self):
        print("="  50)
        print("LOCAL AI CHATBOT")
        print("="  50)

        print("Commands: /reset (reset conversation), /quit (exit)")
        print("="  50)

        while True:
            try:
                userinput = input("\nYou: ").strip()

                if not userinput:

                    continue
                if userinput.lower() == "/quit":
                    print("Goodbye!")
                    break
                if userinput.lower() == "/reset":

                    self.reset()
                    continue

                print("\nAI: ", end="")
                self.chat(userinput)

            except KeyboardInterrupt:
                print("\n\nGoodbye!")
                break

Run chatbot
if name == "main":
    bot = LocalChatbot(
        modelpath="./models/Meta-Llama-3.1-8B-Instruct-Q4KM.gguf",

        systemprompt=(
            "You are an AI assistant for a technology company. "
            "Help users with questions about programming, "
            "data science, and technology in general. "
            "Answer in clear, easy-to-understand language."
        )
    )
    bot.run()

Chatbot with FastAPI Web Interface

from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from llamacpp import Llama

from fastapi.responses import StreamingResponse
import json

app = FastAPI(title="Local LLM Chatbot API")

app.addmiddleware(
    CORSMiddleware,
    alloworigins=[""],

    allowmethods=[""],
    allowheaders=["*"],

)

Load model at startup
llm = Llama(
    modelpath="./models/Meta-Llama-3.1-8B-Instruct-Q4KM.gguf",
    nctx=4096,

    ngpulayers=-1,

    verbose=False
)

class ChatRequest(BaseModel):
    messages: list
    maxtokens: int = 512
    temperature: float = 0.7
    stream: bool = False

@app.post("/v1/chat")
async def chat(request: ChatRequest):
    if request.stream:
        return StreamingResponse(
            streamresponse(request),

            mediatype="text/event-stream"
        )

    response = llm.createchatcompletion(
        messages=request.messages,
        maxtokens=request.maxtokens,
        temperature=request.temperature
    )

    return {
        "response": response["choices"][0]["message"]["content"],
        "usage": response["usage"]
    }

async def streamresponse(request):

    stream = llm.createchatcompletion(

        messages=request.messages,
        maxtokens=request.maxtokens,

        temperature=request.temperature,
        stream=True
    )

    for chunk in stream:
        delta = chunk["choices"][0]["delta"]
        if "content" in delta:
            data = json.dumps({"content": delta["content"]})
            yield f"data: {data}\n\n"

    yield "data: [DONE]\n\n"

@app.get("/health")
async def health():
    return {"status": "ok", "model": "loaded"}

Run with: uvicorn chatbotapi:app --host 0.0.0.0 --port 8000

Optimization Tips and Troubleshooting

Performance Optimization

# 1. Use the right number of threads (usually = number of performance cores)
./llama-cli -m model.gguf -t 8 -p "test"

2. Adjust context length (smaller = faster)
./llama-cli -m model.gguf -c 2048 -p "test"

3. Use optimal batch size
./llama-cli -m model.gguf -b 512 -p "test"

4. Enable memory mapping (on by default)
./llama-cli -m model.gguf --mmap -p "test"

Common Troubleshooting

# Error: "not enough memory"
Solution: Use lower quantization or reduce context length
llm = Llama(
    modelpath="model-Q4KM.gguf",  # Use Q4 instead of Q8

    nctx=2048,                        # Reduce context
    ngpulayers=0                     # CPU only if GPU VRAM is full
)

Error: "CUDA out of memory"
Solution: Offload only some layers to GPU
llm = Llama(
    modelpath="model.gguf",

    ngpulayers=20  # Not -1 (all), just a portion

)

Error: "model file not found"
Solution: Use absolute path
import os
modelpath = os.path.abspath("./models/model.gguf")
llm = Llama(modelpath=modelpath)

Slow performance on CPU
Solution: Recompile with AVX2/AVX512 optimization
make clean && make LLAMAAVX2=1

• Choose the right quantization: Q4KM for the best balance, Q5KM for better quality, Q80 if RAM allows
• Leverage GPU: Offloading to GPU provides significant performance improvements
• Use the OpenAI-compatible server: Makes integration with existing applications easy
• Adjust context length: Smaller context means faster and more memory-efficient inference
• Python bindings: llama-cpp-python makes integration into Python applications straightforward
• Mind RAM requirements: Ensure your hardware matches the chosen model

By mastering llama.cpp, you gain the ability to run powerful AI privately, without API costs, and without rate limits.

References

• llama.cpp Repository: https://github.com/ggerganov/llama.cpp
• llama-cpp-python: https://github.com/abetlen/llama-cpp-python
• HuggingFace GGUF Models: https://huggingface.co/models?library=gguf
• GGUF Specification: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md