Complete txtai Tutorial: All-in-One Embeddings Database for Semantic Search and LLM Workflows

txtai is an open-source Python framework that combines an embeddings database, AI pipelines, and workflow orchestration into a single unified platform. Developed by NeuML, txtai enables developers to build powerful AI applications — from semantic search, question answering, to RAG (Retrieval-Augmented Generation) — without managing multiple separate libraries.

In this tutorial, we will learn how to use txtai comprehensively, from basic installation to advanced usage for building production-ready AI applications.

Why txtai?

Before diving into the technical details, it's important to understand why txtai has become a popular choice:

Installation

Basic Installation

pip install txtai

Installation with Additional Features

txtai has several optional dependencies that can be installed based on your needs:

# Installation with API server support
pip install txtai[api]

Installation with pipeline support (summarization, translation, etc.)
pip install txtai[pipeline]

Installation with all features
pip install txtai[all]

Installation with SQL database support
pip install txtai[database]

Installation with graph support
pip install txtai[graph]

Verify Installation

import txtai
print(f"txtai version: {txtai.version}")

from txtai.embeddings import Embeddings
embeddings = Embeddings()
print("txtai installed successfully!")

Basic Usage: Embeddings Database

Creating a Simple Index

The core concept of txtai is Embeddings — a database that stores and searches data based on semantic similarity, not keyword matching.

from txtai.embeddings import Embeddings

Initialize embeddings with default model
embeddings = Embeddings(path="sentence-transformers/all-MiniLM-L6-v2")

Sample data
data = [
    "Cats are popular pets around the world",
    "Python is a programming language that is easy to learn",
    "Machine learning is changing how we process data",
    "Jakarta is the capital of Indonesia",
    "Deep learning is a subset of machine learning",
    "Dogs are known as man's best friend",
    "JavaScript is used for web development",
    "Natural language processing helps computers understand human language",
    "Bandung is famous for its culinary scene and cool weather",
    "Transfer learning accelerates the AI model training process"
]

Index data
embeddings.index([(i, text, None) for i, text in enumerate(data)])

Search for similar documents
results = embeddings.search("programming language for data science", 3)
for score, idx in results:
    print(f"Score: {score:.4f} | {data[idx]}")

Output:

Score: 0.5823 | Python is a programming language that is easy to learn
Score: 0.4215 | Machine learning is changing how we process data
Score: 0.3891 | JavaScript is used for web development

Indexing with Metadata

txtai also supports storing metadata alongside text:

from txtai.embeddings import Embeddings

embeddings = Embeddings(
    path="sentence-transformers/all-MiniLM-L6-v2",
    content=True  # Enable content storage
)

Data with metadata
documents = [
    {"id": "doc1", "text": "Python installation guide for beginners", "category": "tutorial", "difficulty": "beginner"},
    {"id": "doc2", "text": "PostgreSQL database query optimization", "category": "database", "difficulty": "advanced"},
    {"id": "doc3", "text": "Building REST API with FastAPI", "category": "tutorial", "difficulty": "intermediate"},
    {"id": "doc4", "text": "Introduction to Docker for application deployment", "category": "devops", "difficulty": "beginner"},
    {"id": "doc5", "text": "Microservices architecture with Kubernetes", "category": "devops", "difficulty": "advanced"},
]

Index documents
embeddings.index([(doc["id"], doc) for doc in documents])

Semantic search
results = embeddings.search("how to deploy applications", 3)
for result in results:
    print(f"Score: {result['score']:.4f} | {result['text']} | Category: {result['category']}")

Saving and Loading Index

# Save index to disk
embeddings.save("myindex")


Load index from disk
loadedembeddings = Embeddings()
loadedembeddings.load("myindex")

Use the loaded index
results = loadedembeddings.search("machine learning", 3)

from txtai.embeddings import Embeddings

embeddings = Embeddings(
    path="sentence-transformers/all-MiniLM-L6-v2",
    content=True
)

articles = [
    {"id": 1, "text": "Python tutorial for data analysis", "author": "Ruby", "views": 1500, "category": "python"},
    {"id": 2, "text": "Machine learning with scikit-learn", "author": "Ahmad", "views": 2300, "category": "ml"},
    {"id": 3, "text": "Deep learning using PyTorch", "author": "Ruby", "views": 1800, "category": "ml"},
    {"id": 4, "text": "Web scraping with BeautifulSoup", "author": "Siti", "views": 900, "category": "python"},
    {"id": 5, "text": "Data visualization with Matplotlib", "author": "Ruby", "views": 3200, "category": "python"},
    {"id": 6, "text": "Natural language processing with spaCy", "author": "Ahmad", "views": 1100, "category": "nlp"},
]

embeddings.index([(a["id"], a) for a in articles])

SQL query: find articles by Ruby with views > 1000
results = embeddings.search(
    "SELECT id, text, views FROM txtai WHERE author = 'Ruby' AND views > 1000 ORDER BY views DESC"
)
for result in results:
    print(f"ID: {result['id']} | Views: {result['views']} | {result['text']}")

Combine semantic search with SQL filter
results = embeddings.search(
    "SELECT id, text, score FROM txtai WHERE similar('artificial intelligence') AND category = 'ml' LIMIT 3"
)
for result in results:
    print(f"Score: {result['score']:.4f} | {result['text']}")

from txtai.pipeline import Summary

summary = Summary()

text = """
Artificial intelligence (AI) has experienced rapid development in recent years.
This technology has been applied in various fields, from healthcare, finance, to
transportation. One of the most widely used branches of AI is machine learning,
which enables computers to learn from data without being explicitly programmed.
Deep learning, a subset of machine learning that uses layered neural networks,
has achieved impressive results in image recognition, natural language processing,
and content generation. However, AI development also faces challenges such as
data bias, large computational requirements, and ethical questions surrounding
the use of this technology.
"""

result = summary(text)
print(f"Summary: {result}")

from txtai.pipeline import Translation

translate = Translation()

Translate from English to Spanish
texten = "Machine learning is transforming the way we process information"
result = translate(texten, "en", "es")

print(f"Translation: {result}")

Translate from Spanish to English
textes = "La inteligencia artificial está cambiando el mundo"
result = translate(textes, "es", "en")

print(f"Translation: {result}")

from txtai.pipeline import Extractor

extractor = Extractor(
    embeddings=embeddings,  # Use previously created embeddings
    path="google/flan-t5-base"
)

Context documents
documents = [
    "Python was first released in 1991 by Guido van Rossum",
    "FastAPI is a modern Python web framework released in 2018",
    "Django is a Python web framework first released in 2005",
    "Flask is a micro web framework created by Armin Ronacher",
]

Ask questions
questions = [
    "Who created Python?",
    "When was FastAPI released?",
    "What is Flask?",
]

for question in questions:
    answer = extractor([(question, question, question, False)], documents)
    print(f"Q: {question}")
    print(f"A: {answer[0][1]}\n")

from txtai.pipeline import Labels

labels = Labels()

texts = [
    "BBCA stock price rose 2% today",
    "Indonesia national team won 3-0 against Thailand",
    "A 5.5 magnitude earthquake struck Sulawesi",
    "The latest Marvel movie received high ratings from critics",
]

categories = ["economy", "sports", "disaster", "entertainment"]

for text in texts:
    result = labels(text, categories)
    toplabel = categories[result[0][0]]
    confidence = result[0][1]
    print(f"{text}")
    print(f"  -> Category: {toplabel} (confidence: {confidence:.4f})\n")

from txtai.embeddings import Embeddings
from txtai.pipeline import Extractor

Prepare knowledge base
embeddings = Embeddings(
    path="sentence-transformers/all-MiniLM-L6-v2",
    content=True
)

knowledgebase = [
    {"id": 1, "text": "PT RUBYTHALIB DATA KONSULTA is a data and AI consulting company located in Jakarta."},
    {"id": 2, "text": "The company provides machine learning consulting, data engineering, and AI implementation services."},
    {"id": 3, "text": "Ruby Abdullah is the Director of PT RUBYTHALIB DATA KONSULTA."},
    {"id": 4, "text": "Main services include AI model development, data pipelines, and corporate AI training."},
    {"id": 5, "text": "The office is located in Indonesia Stock Exchange Building, SCBD, South Jakarta."},
]

embeddings.index([(doc["id"], doc) for doc in knowledgebase])


Setup RAG with Extractor
extractor = Extractor(
    embeddings=embeddings,
    path="google/flan-t5-base"
)

Q&A based on knowledge base
questions = [
    "Where is the company office located?",
    "Who is the company director?",
    "What services are offered?",
]

contextdocs = [doc["text"] for doc in knowledgebase]


for q in questions:
    answer = extractor([(q, q, q, False)], contextdocs)
    print(f"Q: {q}")
    print(f"A: {answer[0][1]}\n")

RAG with LLM Integration

txtai supports integration with various LLM providers for more advanced RAG:

from txtai.embeddings import Embeddings
from txtai.pipeline import LLM

Setup embeddings
embeddings = Embeddings(
    path="sentence-transformers/all-MiniLM-L6-v2",
    content=True
)

Knowledge base data
docs = [
    {"id": 1, "text": "Python 3.12 was released in October 2023 with significant performance improvements."},
    {"id": 2, "text": "New features in Python 3.12 include improved error messages and f-string improvements."},
    {"id": 3, "text": "Python 3.12 has 5% better performance compared to Python 3.11."},
    {"id": 4, "text": "New type parameter syntax was introduced in Python 3.12 for generic classes."},
]

embeddings.index([(d["id"], d) for d in docs])

Setup LLM
llm = LLM("google/flan-t5-base")

RAG: Retrieve relevant docs, then generate answer
query = "What's new in Python 3.12?"
results = embeddings.search(query, 3)

Combine context
context = "\n".join([r["text"] for r in results])
prompt = f"""Based on the following information:
{context}

Answer the question: {query}
Answer:"""

answer = llm(prompt)
print(f"Q: {query}")
print(f"A: {answer}")

Workflows: Pipeline Orchestration

Workflows allow you to combine multiple pipelines into an integrated processing flow:

from txtai.embeddings import Embeddings
from txtai.pipeline import Summary, Translation, Labels
from txtai.workflow import Workflow, Task

Initialize pipelines
summary = Summary()
translate = Translation()

Define workflow
workflow = Workflow([
    Task(lambda x: summary(x, maxlength=50)),    # Step 1: Summarize text
    Task(lambda x: translate(x, "en", "es")),     # Step 2: Translate to Spanish
])

Run workflow
texts = [
    """Artificial intelligence has made significant progress in recent years.
    Large language models can now generate human-like text, write code, and
    answer complex questions. These advances are transforming industries
    from healthcare to finance.""",
    """Cloud computing enables businesses to scale their infrastructure
    on demand. Major providers like AWS, Google Cloud, and Azure offer
    comprehensive services for computing, storage, and machine learning."""
]

results = list(workflow(texts))
for i, result in enumerate(results):
    print(f"Result {i+1}: {result}\n")

API Server

txtai provides a built-in API server that can be used directly for serving:

YAML Configuration

Create a config.yml file:

# Embeddings configuration
path: sentence-transformers/all-MiniLM-L6-v2
content: true

Writable index
writable: true

Running the API Server

# Install API dependency
pip install txtai[api]

Run server
CONFIG=config.yml uvicorn "txtai.api:app" --host 0.0.0.0 --port 8080

Using the API

import requests

BASEURL = "http://localhost:8080"


Add documents
requests.post(f"{BASEURL}/add", json=[
    {"id": "1", "text": "Python tutorial for beginners"},
    {"id": "2", "text": "Machine learning with scikit-learn"},
    {"id": "3", "text": "Deep learning using TensorFlow"},
])

Index documents
requests.get(f"{BASEURL}/index")


Search documents
response = requests.get(f"{BASEURL}/search", params={"query": "learn AI", "limit": 3})
results = response.json()
for r in results:
    print(f"Score: {r['score']:.4f} | {r['text']}")

API Server with Docker

FROM python:3.11-slim

WORKDIR /app
RUN pip install txtai[api]

COPY config.yml .

EXPOSE 8080
CMD ["uvicorn", "txtai.api:app", "--host", "0.0.0.0", "--port", "8080"]

docker build -t txtai-api .
docker run -p 8080:8080 -v $(pwd)/config.yml:/app/config.yml txtai-api

Hybrid Search

txtai supports hybrid search that combines semantic search with keyword search (BM25):

from txtai.embeddings import Embeddings

embeddings = Embeddings(
    path="sentence-transformers/all-MiniLM-L6-v2",
    content=True,
    hybrid=True,  # Enable hybrid search
    scoring={
        "method": "bm25",
        "terms": True
    }
)

documents = [
    {"id": 1, "text": "Tutorial on using Pandas library for tabular data analysis"},
    {"id": 2, "text": "Pandas are rare animals protected in China"},
    {"id": 3, "text": "DataFrame manipulation with Pandas: groupby, merge, and pivot"},
    {"id": 4, "text": "Reading CSV and Excel files using Python Pandas library"},
    {"id": 5, "text": "Giant panda habitat in the Sichuan mountains is threatened by deforestation"},
]

embeddings.index([(d["id"], d) for d in documents])

Semantic search alone might confuse Pandas (library) vs Panda (animal)
Hybrid search is more accurate because it combines semantic context + keyword matching
results = embeddings.search("Pandas Python library data analysis", 3)
for r in results:
    print(f"Score: {r['score']:.4f} | {r['text']}")

Graph Database

txtai also supports graph database for storing relationships between documents:

from txtai.embeddings import Embeddings

embeddings = Embeddings(
    path="sentence-transformers/all-MiniLM-L6-v2",
    content=True,
    graph={
        "approximate": False,
        "topics": {}
    }
)

documents = [
    {"id": 1, "text": "Python is a popular programming language for data science"},
    {"id": 2, "text": "Pandas library is used for data manipulation in Python"},
    {"id": 3, "text": "NumPy provides numerical array operations for Python"},
    {"id": 4, "text": "Scikit-learn is a Python and NumPy-based machine learning library"},
    {"id": 5, "text": "TensorFlow is a deep learning framework from Google"},
    {"id": 6, "text": "PyTorch is a deep learning framework from Meta"},
]

embeddings.index([(d["id"], d) for d in documents])

Query graph to find connected documents
graph = embeddings.graph
print(f"Total nodes: {graph.count()}")

Find documents connected to a specific topic
results = embeddings.search("Python data science ecosystem", 5)
for r in results:
    print(f"Score: {r['score']:.4f} | {r['text']}")

Best Practices

1. Choose the Right Model

# For English - fast and efficient
embeddings = Embeddings(path="sentence-transformers/all-MiniLM-L6-v2")

For higher quality
embeddings = Embeddings(path="sentence-transformers/all-mpnet-base-v2")

For multilingual (including Bahasa Indonesia)
embeddings = Embeddings(path="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")

For specific domains, use fine-tuned models
embeddings = Embeddings(path="your-org/custom-finetuned-model")

2. Optimize Indexing for Large Datasets

from txtai.embeddings import Embeddings

embeddings = Embeddings(
    path="sentence-transformers/all-MiniLM-L6-v2",
    content=True,
    batch=32,       # Batch size for encoding
    encodebatch=128, # Batch size for index encoding
    faiss={
        "components": "IVF100,Flat",  # Use IVF for large datasets
        "nprobe": 10
    }
)

For very large datasets, use incremental upsert
def indexinbatches(embeddings, data, batchsize=1000):

    for i in range(0, len(data), batchsize):
        batch = data[i:i + batchsize]

        embeddings.upsert([(item["id"], item) for item in batch])
        print(f"Indexed {min(i + batchsize, len(data))}/{len(data)} documents")

indexinbatches(embeddings, largedataset)

# Save to disk
embeddings.save("/path/to/index")

Load back
embeddings = Embeddings()
embeddings.load("/path/to/index")

Use cloud storage (S3, GCS)
embeddings.save("s3://bucket/path/to/index")
embeddings.load("s3://bucket/path/to/index")

import logging
import time

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("txtai")

def timedsearch(embeddings, query, limit=5):
    start = time.time()
    results = embeddings.search(query, limit)
    elapsed = time.time() - start
    logger.info(f"Search '{query}' returned {len(results)} results in {elapsed:.3f}s")
    return results

5. Proper Error Handling

from txtai.embeddings import Embeddings

def safeindexandsearch(data, query):

    try:
        embeddings = Embeddings(
            path="sentence-transformers/all-MiniLM-L6-v2",
            content=True
        )
        embeddings.index([(d["id"], d) for d in data])
        results = embeddings.search(query, 5)
        return results
    except FileNotFoundError:
        print("Model not found. Make sure the model has been downloaded.")
        return []
    except Exception as e:
        print(f"Error: {e}")
        return []

from txtai.embeddings import Embeddings
from txtai.pipeline import Summary
import json

class DocSearchEngine:
    def init(self, model="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"):
        self.embeddings = Embeddings(
            path=model,
            content=True,
            hybrid=True,
            scoring={"method": "bm25", "terms": True}
        )
        self.summary = Summary()
        self.doccount = 0

    def adddocuments(self, documents):

        """Add documents to the index."""
        indexed = []
        for doc in documents:
            self.doccount += 1
            indexed.append((self.doccount, {

                "text": doc["content"],
                "title": doc.get("title", ""),
                "source": doc.get("source", ""),
                "category": doc.get("category", "general"),
            }))
        self.embeddings.index(indexed)
        print(f"Total indexed documents: {self.doccount}")

    def search(self, query, limit=5, category=None):
        """Search documents by query."""
        if category:
            sql = f"SELECT id, text, title, source, score FROM txtai WHERE similar('{query}') AND category = '{category}' LIMIT {limit}"
        else:
            sql = f"SELECT id, text, title, source, score FROM txtai WHERE similar('{query}') LIMIT {limit}"

        return self.embeddings.search(sql)

    def summarizeresults(self, results):

        """Summarize search results."""
        combined = " ".join([r["text"] for r in results[:3]])
        return self.summary(combined, maxlength=100)

    def save(self, path):
        self.embeddings.save(path)

    def load(self, path):
        self.embeddings.load(path)


Usage
engine = DocSearchEngine()

docs = [
    {"title": "Python Installation", "content": "Python can be installed via python.org or using pyenv for version management.", "category": "setup"},
    {"title": "Virtual Environment", "content": "Use venv or conda to create isolated Python environments.", "category": "setup"},
    {"title": "Pandas Basics", "content": "Pandas provides DataFrame and Series for efficient tabular data manipulation.", "category": "data"},
    {"title": "Model Training", "content": "Training machine learning models involves algorithm selection, hyperparameter tuning, and evaluation.", "category": "ml"},
    {"title": "Model Deployment", "content": "Models can be deployed using FastAPI, Flask, or managed platforms like SageMaker.", "category": "ml"},
]

engine.adddocuments(docs)

results = engine.search("how to setup Python project", limit=3, category="setup")
for r in results:
    print(f"[{r['title']}] Score: {r['score']:.4f}")
    print(f"  {r['text']}\n")

summary = engine.summarize_results(results)
print(f"Summary: {summary}")

Conclusion

txtai is an extremely powerful framework for building AI applications based on semantic search and embeddings. Its key strengths are:

• Simplicity: Clean and easy-to-use API, suitable for both rapid prototyping and production
• Versatility: Combines embeddings database, NLP pipelines, and workflows in a single platform
• SQL Support: The ability to run SQL queries on vector databases makes data filtering and analysis extremely flexible
• Hybrid Search: Combining semantic and keyword search delivers more accurate search results
• API Server: Built-in server makes deployment easy without writing boilerplate code

To get started, simply install txtai, create an embeddings index from your data, and start experimenting with semantic search. From there, you can add NLP pipelines, workflow orchestration, and other advanced features as needed.

Official txtai documentation is available at the NeuML/txtai GitHub repository and is very comprehensive with examples that can be practiced directly.