LocalAI: Run AI Models Locally with an OpenAI-Compatible API
Author: Ruby Abdullah — PT RUBYTHALIB DATA KONSULTA Date: November 7, 2026 Category: Tutorial Tags: LocalAI, LLM, Self-Hosted AI, OpenAI-Compatible, DockerIntroduction
Privacy concerns, API costs, and latency are three of the biggest pain points when building AI-powered applications with cloud-based providers. Every request you send to OpenAI or Anthropic travels over the internet, is logged by a third party, and costs money. For sensitive data — medical records, financial documents, internal company knowledge — this is often unacceptable.
LocalAI solves this problem. It is a free, open-source drop-in replacement for the OpenAI API that runs entirely on your own hardware. You point your existing OpenAI SDK code at http://localhost:8080 instead of https://api.openai.com, and it just works. No API keys, no usage fees, no data leaving your server.
LocalAI supports a wide range of capabilities:
- Text generation using LLMs like LLaMA, Mistral, Phi, Gemma, and many others
- Embeddings for semantic search and RAG pipelines
- Image generation via Stable Diffusion
- Speech-to-text (Whisper) and text-to-speech (Piper)
- Function calling and tool use
- Vision (multimodal image understanding)
The project is built on top of llama.cpp and other inference backends, meaning it can run efficiently on consumer hardware — even without a GPU, using CPU quantized models.
This tutorial walks through setting up LocalAI from scratch, downloading models, using the core API endpoints, and integrating everything with Python. By the end, you will have a fully working local AI stack that mirrors the OpenAI API surface.
Prerequisites
Before starting, ensure your system meets these requirements:
- OS: Linux, macOS, or Windows (WSL2 recommended on Windows)
- RAM: At least 8 GB (16 GB recommended for 7B parameter models)
- Disk: At least 10 GB free space for model files
- Docker: Docker Engine 20.10+ and Docker Compose v2
- Python: 3.9+ with pip
For GPU acceleration (optional but recommended):
- NVIDIA GPU with CUDA 11.8+ drivers
- Or Apple Silicon Mac for Metal acceleration
Installation and Setup
Option 1: Docker (Recommended)
Docker is the fastest way to get LocalAI running. The official Docker image bundles all dependencies and backends.
Pull and run with CPU support:docker run -p 8080:8080 \
-v $HOME/.local/share/localai/models:/build/models \
-e DEBUG=true \
localai/localai:latest-aio-cpu
The -v flag mounts a local directory for model storage so models persist between container restarts.
docker run --gpus all -p 8080:8080 \
-v $HOME/.local/share/localai/models:/build/models \
-e DEBUG=true \
localai/localai:latest-aio-gpu-nvidia-cuda-12
Using Docker Compose (recommended for production):
Create a docker-compose.yml file:
version: "3.8"
services:
localai:
image: localai/localai:latest-aio-cpu
ports:
- "8080:8080"
volumes:
- ./models:/build/models
environment:
- DEBUG=false
- THREADS=4
- CONTEXTSIZE=4096
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
interval: 30s
timeout: 10s
retries: 5
Start the service:
docker compose up -d
Check that LocalAI is running:
curl http://localhost:8080/readyz
Expected: {"status":"ok"}
Option 2: Binary Installation
For systems where Docker is not available, download the pre-built binary:
# Download the latest release (adjust version and platform)
curl -Lo localai https://github.com/mudler/LocalAI/releases/download/v2.20.0/local-ai-Linux-x8664
chmod +x localai
Run LocalAI
./localai run --models-path ./models --address :8080
Option 3: Build from Source
git clone https://github.com/mudler/LocalAI.git
cd LocalAI
Build with CUDA support
BUILDTYPE=cublas make build
Or build for CPU only
make build
./local-ai run --models-path ./models
Downloading and Configuring Models
LocalAI uses GGUF format models from Hugging Face. The easiest way to get started is using the built-in model gallery.
Using the Model Gallery
LocalAI has a gallery of pre-configured models you can install with a single API call:
# List available models in the gallery
curl http://localhost:8080/models/available | python3 -m json.tool | head -50
Install a model from the gallery (e.g., Mistral 7B)
curl http://localhost:8080/models/apply -H "Content-Type: application/json" -d '{
"id": "huggingface@TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4KM.gguf"
}'
Manually Downloading GGUF Models
Download a quantized model from Hugging Face directly into your models directory:
mkdir -p $HOME/.local/share/localai/models
Download Mistral 7B Q4 quantization (~4 GB)
wget -O $HOME/.local/share/localai/models/mistral-7b-instruct.gguf \
https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/resolve/main/mistral-7b-instruct-v0.2.Q4KM.gguf
Download a small embedding model (~67 MB)
wget -O $HOME/.local/share/localai/models/all-minilm-l6-v2.bin \
https://huggingface.co/mudler/all-MiniLM-L6-v2/resolve/main/ggml-model-q40.bin
Creating Model Configuration Files
LocalAI uses YAML configuration files to tell it how to serve each model. Create a config file for your LLM:
# models/mistral-7b-instruct.yaml
name: mistral-7b-instruct
parameters:
model: mistral-7b-instruct.gguf
temperature: 0.7
topp: 0.9
topk: 40
maxnewtokens: 2048
contextsize: 4096
threads: 4
f16: true
mmap: true
template:
chatmessage: |
{{if eq .RoleName "system"}}[INST] {{.Content}} [/INST]{{end}}
{{if eq .RoleName "user"}}[INST] {{.Content}} [/INST]{{end}}
{{if eq .RoleName "assistant"}}{{.Content}}{{end}}
For an embedding model:
# models/text-embedding-ada-002.yaml
name: text-embedding-ada-002
parameters:
model: all-minilm-l6-v2.bin
backend: llama
embeddings: true
After adding these files, restart LocalAI and verify the models are loaded:
curl http://localhost:8080/v1/models | python3 -m json.tool
Basic Usage
Text Generation
LocalAI exposes the same /v1/chat/completions endpoint as OpenAI. You can call it directly with curl:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "mistral-7b-instruct",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain what machine learning is in 3 sentences."}
],
"temperature": 0.7,
"maxtokens": 256
}'
For streaming responses (token by token output), add "stream": true:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "mistral-7b-instruct",
"messages": [{"role": "user", "content": "Write a haiku about data engineering."}],
"stream": true
}'
Generating Embeddings
Embeddings are vector representations of text, used for semantic search, clustering, and RAG systems:
curl http://localhost:8080/v1/embeddings \
-H "Content-Type: application/json" \
-d '{
"model": "text-embedding-ada-002",
"input": "LocalAI is a self-hosted alternative to the OpenAI API"
}'
The response contains a data[0].embedding array — a list of floating-point numbers representing the semantic meaning of the input text.
Python Integration
This is where LocalAI becomes truly powerful. Because it mirrors the OpenAI API, you can use the official openai Python SDK with zero modifications to your code — just change the base URL.
Install the OpenAI SDK
pip install openai
Basic Client Setup
from openai import OpenAI
Point the client at your local LocalAI instance
client = OpenAI(
baseurl="http://localhost:8080/v1",
apikey="not-needed" # LocalAI does not require an API key by default
)
Test the connection
models = client.models.list()
for model in models.data:
print(f"Available model: {model.id}")
Text Generation with Chat Completions
from openai import OpenAI
client = OpenAI(
baseurl="http://localhost:8080/v1",
apikey="not-needed"
)
def asklocalai(question: str, systemprompt: str = "You are a helpful assistant.") -> str:
response = client.chat.completions.create(
model="mistral-7b-instruct",
messages=[
{"role": "system", "content": systemprompt},
{"role": "user", "content": question}
],
temperature=0.7,
maxtokens=512
)
return response.choices[0].message.content
Example usage
answer = asklocalai("What are the benefits of running AI models locally?")
print(answer)
Streaming Responses
Streaming allows you to display text as it is generated rather than waiting for the complete response:
from openai import OpenAI
client = OpenAI(
baseurl="http://localhost:8080/v1",
apikey="not-needed"
)
def streamresponse(prompt: str):
stream = client.chat.completions.create(
model="mistral-7b-instruct",
messages=[{"role": "user", "content": prompt}],
stream=True,
maxtokens=1024
)
print("Response: ", end="", flush=True)
for chunk in stream:
delta = chunk.choices[0].delta
if delta.content:
print(delta.content, end="", flush=True)
print() # newline at end
streamresponse("Write a step-by-step guide to making nasi goreng.")
Generating and Using Embeddings
from openai import OpenAI
import numpy as np
client = OpenAI(
baseurl="http://localhost:8080/v1",
apikey="not-needed"
)
def getembedding(text: str) -> list[float]:
response = client.embeddings.create(
model="text-embedding-ada-002",
input=text
)
return response.data[0].embedding
def cosinesimilarity(veca: list[float], vecb: list[float]) -> float:
a = np.array(veca)
b = np.array(vecb)
return float(np.dot(a, b) / (np.linalg.norm(a) np.linalg.norm(b)))
Simple semantic search example
documents = [
"LocalAI provides a self-hosted OpenAI-compatible API",
"Docker is a containerization platform for running applications",
"Machine learning models learn patterns from training data",
"Python is a popular language for data science and AI",
"LocalAI supports LLMs, image generation, and speech recognition"
]
query = "How can I run AI without sending data to the cloud?"
Generate embeddings
queryembedding = getembedding(query)
docembeddings = [getembedding(doc) for doc in documents]
Find most relevant documents
scores = [(doc, cosinesimilarity(queryembedding, emb))
for doc, emb in zip(documents, docembeddings)]
scores.sort(key=lambda x: x[1], reverse=True)
print(f"Query: {query}\n")
print("Most relevant documents:")
for doc, score in scores[:3]:
print(f" [{score:.3f}] {doc}")
Advanced Usage
Function Calling
Function calling allows the model to request structured data from your application, enabling it to call tools, fetch information, or trigger actions. LocalAI supports this feature with compatible models.
from openai import OpenAI
import json
client = OpenAI(
baseurl="http://localhost:8080/v1",
apikey="not-needed"
)
Define tools the model can use
tools = [
{
"type": "function",
"function": {
"name": "getweather",
"description": "Get the current weather for a given city",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "The city name, e.g. Jakarta"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["city"]
}
}
}
]
Simulated tool execution
def getweather(city: str, unit: str = "celsius") -> dict:
# In a real app, this would call a weather API
return {
"city": city,
"temperature": 32 if unit == "celsius" else 89,
"unit": unit,
"condition": "Partly cloudy",
"humidity": "78%"
}
def runconversation(usermessage: str):
messages = [{"role": "user", "content": usermessage}]
# First call — model may request a tool
response = client.chat.completions.create(
model="mistral-7b-instruct",
messages=messages,
tools=tools,
toolchoice="auto"
)
choice = response.choices[0]
# If the model wants to call a function
if choice.finishreason == "toolcalls":
toolcall = choice.message.toolcalls[0]
functionname = toolcall.function.name
arguments = json.loads(toolcall.function.arguments)
print(f"Model called function: {functionname}({arguments})")
# Execute the function
result = getweather(arguments)
# Add the tool result to the conversation
messages.append(choice.message)
messages.append({
"role": "tool",
"toolcallid": toolcall.id,
"content": json.dumps(result)
})
# Second call — model uses the tool result
finalresponse = client.chat.completions.create(
model="mistral-7b-instruct",
messages=messages
)
return finalresponse.choices[0].message.content
return choice.message.content
result = runconversation("What is the weather like in Jakarta right now?")
print(result)
Vision: Image Understanding
LocalAI supports vision models that can analyze images. You need a multimodal model such as LLaVA.
First, download a vision model:
wget -O $HOME/.local/share/localai/models/llava-v1.6-mistral-7b.Q4KM.gguf \
https://huggingface.co/cjpais/llava-1.6-mistral-7b-gguf/resolve/main/llava-1.6-mistral-7b.Q4
KM.gguf
wget -O $HOME/.local/share/localai/models/llava-v1.6-mistral-7b.mmproj.gguf \
https://huggingface.co/cjpais/llava-1.6-mistral-7b-gguf/resolve/main/mmproj-model-f16.gguf
Create a config file for the vision model:
# models/llava.yaml
name: llava
parameters:
model: llava-v1.6-mistral-7b.Q4KM.gguf
mmproj: llava-v1.6-mistral-7b.mmproj.gguf
contextsize: 4096
threads: 4
Use it with Python:
from openai import OpenAI
import base64
from pathlib import Path
client = OpenAI(
baseurl="http://localhost:8080/v1",
apikey="not-needed"
)
def analyzeimagefromfile(imagepath: str, question: str) -> str:
# Read and encode the image as base64
imagedata = Path(imagepath).readbytes()
encoded = base64.b64encode(imagedata).decode("utf-8")
response = client.chat.completions.create(
model="llava",
messages=[
{
"role": "user",
"content": [
{
"type": "imageurl",
"imageurl": {
"url": f"data:image/jpeg;base64,{encoded}"
}
},
{
"type": "text",
"text": question
}
]
}
],
maxtokens=512
)
return response.choices[0].message.content
def analyzeimagefromurl(imageurl: str, question: str) -> str:
response = client.chat.completions.create(
model="llava",
messages=[
{
"role": "user",
"content": [
{
"type": "imageurl",
"imageurl": {"url": imageurl}
},
{
"type": "text",
"text": question
}
]
}
],
maxtokens=512
)
return response.choices[0].message.content
Analyze an image from URL
result = analyzeimagefromurl(
"https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNGtransparencydemonstration1.png/280px-PNGtransparencydemonstration1.png",
"Describe what you see in this image."
)
print(result)
Speech Recognition (Whisper)
LocalAI includes OpenAI Whisper for transcribing audio to text:
# Download the Whisper model
wget -O $HOME/.local/share/localai/models/whisper-1 \
https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin
from openai import OpenAI
client = OpenAI(
baseurl="http://localhost:8080/v1",
apikey="not-needed"
)
def transcribeaudio(audiofilepath: str) -> str:
with open(audiofilepath, "rb") as audiofile:
transcription = client.audio.transcriptions.create(
model="whisper-1",
file=audiofile,
responseformat="text"
)
return transcription
Usage
text = transcribeaudio("meetingrecording.mp3")
print(f"Transcription: {text}")
Text-to-Speech
LocalAI supports text-to-speech via Piper or other TTS backends:
# Via curl
curl http://localhost:8080/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"model": "tts-1",
"input": "Hello from LocalAI! This is a test of text to speech.",
"voice": "alloy"
}' \
--output speech.mp3
from openai import OpenAI
from pathlib import Path
client = OpenAI(
baseurl="http://localhost:8080/v1",
apikey="not-needed"
)
def texttospeech(text: str, outputpath: str = "output.mp3"):
response = client.audio.speech.create(
model="tts-1",
voice="alloy",
input=text
)
response.streamtofile(outputpath)
print(f"Audio saved to {outputpath}")
texttospeech("LocalAI enables private, local AI inference for everyone.")
Building a Simple RAG Pipeline
Here is a practical end-to-end example combining embeddings and chat completion to build a simple Retrieval-Augmented Generation (RAG) system:
from openai import OpenAI
import numpy as np
from typing import List
client = OpenAI(
baseurl="http://localhost:8080/v1",
apikey="not-needed"
)
class SimpleRAG:
def init(self):
self.documents: List[str] = []
self.embeddings: List[List[float]] = []
def adddocument(self, text: str):
"""Add a document to the knowledge base."""
embedding = self.embed(text)
self.documents.append(text)
self.embeddings.append(embedding)
print(f"Added document: {text[:60]}...")
def embed(self, text: str) -> List[float]:
response = client.embeddings.create(
model="text-embedding-ada-002",
input=text
)
return response.data[0].embedding
def cosinesimilarity(self, a: List[float], b: List[float]) -> float:
va, vb = np.array(a), np.array(b)
return float(np.dot(va, vb) / (np.linalg.norm(va) np.linalg.norm(vb)))
def retrieve(self, query: str, topk: int = 3) -> List[str]:
"""Retrieve the most relevant documents for a query."""
queryemb = self.embed(query)
scores = [
(doc, self.cosinesimilarity(queryemb, emb))
for doc, emb in zip(self.documents, self.embeddings)
]
scores.sort(key=lambda x: x[1], reverse=True)
return [doc for doc, in scores[:topk]]
def answer(self, question: str) -> str:
"""Answer a question using retrieved context."""
relevantdocs = self.retrieve(question)
context = "\n\n".join(f"- {doc}" for doc in relevantdocs)
prompt = f"""Answer the question based only on the provided context.
Context:
{context}
Question: {question}
Answer:"""
response = client.chat.completions.create(
model="mistral-7b-instruct",
messages=[
{"role": "system", "content": "You are a helpful assistant. Answer only based on the provided context."},
{"role": "user", "content": prompt}
],
temperature=0.3,
maxtokens=512
)
return response.choices[0].message.content
Example usage
rag = SimpleRAG()
Add knowledge base documents
rag.adddocument("LocalAI is an open-source, self-hosted alternative to OpenAI API that runs locally.")
rag.adddocument("LocalAI supports LLM inference, image generation, speech-to-text, and text-to-speech.")
rag.adddocument("LocalAI uses GGUF format models from Hugging Face and llama.cpp as the inference backend.")
rag.adddocument("Docker is the recommended way to run LocalAI for easy setup and dependency management.")
rag.adddocument("LocalAI can run on CPU without a GPU, though GPU acceleration significantly improves speed.")
rag.adddocument("PT RUBYTHALIB DATA KONSULTA provides AI consulting and development services in Indonesia.")
Ask questions
questions = [
"What file format does LocalAI use for models?",
"Can LocalAI run without a GPU?",
"What services does PT RUBYTHALIB DATA KONSULTA offer?"
]
for q in questions:
print(f"\nQ: {q}")
print(f"A: {rag.answer(q)}")
Best Practices
1. Choose the Right Quantization Level
GGUF models come in different quantization levels. This affects quality, speed, and RAM usage:
| Quantization | Size (7B model) | RAM Usage | Quality |
|---|---|---|---|
| Q2K | ~2.7 GB | ~5 GB | Poor |
| Q4KM | ~4.1 GB | ~6.5 GB | Good (recommended) |
| Q5KM | ~4.8 GB | ~7.5 GB | Very good |
| Q80 | ~7.2 GB | ~9.5 GB | Near-original |
| F16 | ~14 GB | ~16 GB | Original |
For most use cases, Q4KM offers the best balance of quality and resource usage.
2. Configure Thread Count Appropriately
LocalAI uses CPU threads for inference. Set the thread count to your physical core count (not logical/hyperthreaded):
# In your model YAML config
threads: 8 # Set to number of physical CPU cores
Or set via environment variable:
docker run -e THREADS=8 localai/localai:latest-aio-cpu
3. Enable Memory Mapping
Memory mapping (mmap) allows the OS to handle model loading efficiently and reduces RAM usage on large models:
mmap: true
mlock: false # Set to true to pin model to RAM and prevent swapping
4. Use Context Caching for Multi-turn Conversations
Keep conversation history in your application rather than re-sending system prompts each time. LocalAI caches the KV state for repeated prompt prefixes, which speeds up inference:
# Good practice: maintain message history
class ConversationManager:
def init(self, systemprompt: str):
self.messages = [{"role": "system", "content": systemprompt}]
def chat(self, usermessage: str) -> str:
self.messages.append({"role": "user", "content": usermessage})
response = client.chat.completions.create(
model="mistral-7b-instruct",
messages=self.messages,
maxtokens=512
)
assistantmessage = response.choices[0].message.content
self.messages.append({"role": "assistant", "content": assistantmessage})
return assistantmessage
def reset(self):
system = self.messages[0]
self.messages = [system] # Keep system prompt, clear history
5. Implement Retry Logic for Production
LocalAI can occasionally time out under heavy load. Add retry logic for robust applications:
import time
from openai import OpenAI, APITimeoutError, APIConnectionError
client = OpenAI(
baseurl="http://localhost:8080/v1",
apikey="not-needed",
timeout=120.0 # Increase timeout for large models
)
def robustcompletion(messages: list, retries: int = 3, delay: float = 2.0) -> str:
for attempt in range(retries):
try:
response = client.chat.completions.create(
model="mistral-7b-instruct",
messages=messages,
maxtokens=512
)
return response.choices[0].message.content
except (APITimeoutError, APIConnectionError) as e:
if attempt < retries - 1:
print(f"Attempt {attempt + 1} failed: {e}. Retrying in {delay}s...")
time.sleep(delay)
delay *= 2 # Exponential backoff
else:
raise
return ""
6. Monitor Resource Usage
Watch memory and CPU usage, especially with larger models:
# Monitor Docker container resources
docker stats localai
Check if model is fully loaded (no swap usage)
free -h
Monitor with htop (CPU + memory per process)
htop
7. Security Considerations
LocalAI by default has no authentication. For anything beyond local development:
# Enable API key authentication
docker run -p 8080:8080 \
-e APIKEY=your-secret-key \
-v ./models:/build/models \
localai/localai:latest-aio-cpu
# Use the API key in your Python client
client = OpenAI(
baseurl="http://your-server:8080/v1",
apikey="your-secret-key"
)
For production deployments, additionally:
- Put LocalAI behind a reverse proxy (Nginx, Traefik) with TLS
- Restrict network access with firewall rules
- Never expose LocalAI directly to the internet without authentication
8. Optimizing for Specific Use Cases
For RAG / Q&A systems: Use smaller, faster models (Phi-3 Mini, Gemma 2B) since the context is provided. Accuracy matters more than creativity. For code generation: Try CodeLLaMA or DeepSeek Coder models which are specifically trained on code. For conversational chat: Mistral 7B Instruct or LLaMA 3.1 8B Instruct offer an excellent balance. For summarization: Smaller 3B-7B models work well and are much faster than 13B+ models.Troubleshooting Common Issues
Model not loading:# Check LocalAI logs
docker logs localai --tail 50
Verify model file exists and is readable
ls -lh $HOME/.local/share/localai/models/
Out of memory errors:
- Switch to a lower quantization (Q4KM instead of Q80)
- Reduce context size in the model YAML config
- Try a smaller parameter model (3B or 1B)
- Ensure
mmap: trueis set in model config - Increase thread count to match physical CPU cores
- Consider GPU acceleration if available
- Use quantized models (Q4 is 2-3x faster than F16)
# Check if container is running
docker ps
Check if port is exposed correctly
curl -v http://localhost:8080/readyz
Conclusion
LocalAI delivers on its core promise: a fully OpenAI-compatible inference server that runs entirely on your own hardware. The migration path for existing OpenAI-based applications is remarkably simple — in most cases, just two lines of code change (the base URL and the API key).
The practical benefits are significant for businesses handling sensitive data:
- Privacy: Your documents and queries never leave your infrastructure
- Cost control: No per-token charges, predictable server costs
- Latency: Local inference eliminates round-trip network latency
- Reliability: No dependency on external API availability
- Compliance: Easier to meet data residency and privacy regulations
At PT RUBYTHALIB DATA KONSULTA, we use LocalAI in projects where client data must remain within their own infrastructure — financial document analysis, HR systems with employee data, and internal knowledge bases. The OpenAI-compatible API means we can prototype with OpenAI, then deploy with LocalAI for production without rewriting client code.
The main trade-off is hardware. A 7B parameter model on a CPU will be significantly slower than the OpenAI API, often taking 5-30 seconds for a medium-length response depending on your hardware. With a modern NVIDIA GPU (RTX 3090 or better), inference speeds become comparable to hosted APIs.
For teams getting started: begin with Docker, use a Q4K_M quantized Mistral 7B or Phi-3 model, and build your application against the standard OpenAI SDK. You will have a working local AI stack in under an hour that you can iterate on and eventually deploy to production with confidence.
The LocalAI project is actively maintained and the ecosystem of supported models and backends continues to grow. It represents a practical, production-ready solution for self-hosted AI inference today.
Resources:- LocalAI GitHub: https://github.com/mudler/LocalAI
- LocalAI Documentation: https://localai.io
- Hugging Face GGUF Models: https://huggingface.co/models?library=gguf
- llama.cpp (inference backend): https://github.com/ggerganov/llama.cpp