Complete txtai Tutorial: All-in-One Embeddings Database for Semantic Search and LLM Workflows

txtai is an open-source Python framework that combines an embeddings database, AI pipelines, and workflow orchestration into a single unified platform. Developed by NeuML, txtai enables developers to build powerful AI applications — from semantic search, question answering, to RAG (Retrieval-Augmented Generation) — without managing multiple separate libraries.

In this tutorial, we will learn how to use txtai comprehensively, from basic installation to advanced usage for building production-ready AI applications.

Why txtai?

Before diving into the technical details, it's important to understand why txtai has become a popular choice:

All-in-One: Combines embeddings database, NLP pipelines, and workflow orchestration in a single package

Easy to Use: Intuitive API — you can get started with just a few lines of code

Flexible: Supports various models from Hugging Face, sentence-transformers, and custom models

Lightweight: No external database server required — everything runs in-process

Production-Ready: Supports API server, persistent indexing, and integration with various data formats

Installation

Basic Installation

pip install txtai

Installation with Additional Features

txtai has several optional dependencies that can be installed based on your needs:

# Installation with API server support
pip install txtai[api]

Installation with pipeline support (summarization, translation, etc.)
pip install txtai[pipeline]

Installation with all features
pip install txtai[all]

Installation with SQL database support
pip install txtai[database]

Installation with graph support
pip install txtai[graph]

Verify Installation

import txtai
print(f"txtai version: {txtai.version}")

from txtai.embeddings import Embeddings
embeddings = Embeddings()
print("txtai installed successfully!")

Basic Usage: Embeddings Database

Creating a Simple Index

The core concept of txtai is Embeddings — a database that stores and searches data based on semantic similarity, not keyword matching.

from txtai.embeddings import Embeddings

Initialize embeddings with default model
embeddings = Embeddings(path="sentence-transformers/all-MiniLM-L6-v2")

Sample data
data = [
    "Cats are popular pets around the world",
    "Python is a programming language that is easy to learn",
    "Machine learning is changing how we process data",
    "Jakarta is the capital of Indonesia",
    "Deep learning is a subset of machine learning",
    "Dogs are known as man's best friend",
    "JavaScript is used for web development",
    "Natural language processing helps computers understand human language",
    "Bandung is famous for its culinary scene and cool weather",
    "Transfer learning accelerates the AI model training process"
]

Index data
embeddings.index([(i, text, None) for i, text in enumerate(data)])

Search for similar documents
results = embeddings.search("programming language for data science", 3)
for score, idx in results:
    print(f"Score: {score:.4f} | {data[idx]}")

Output:

Score: 0.5823 | Python is a programming language that is easy to learn
Score: 0.4215 | Machine learning is changing how we process data
Score: 0.3891 | JavaScript is used for web development

Indexing with Metadata

txtai also supports storing metadata alongside text:

from txtai.embeddings import Embeddings

embeddings = Embeddings(
    path="sentence-transformers/all-MiniLM-L6-v2",
    content=True  # Enable content storage
)

Data with metadata
documents = [
    {"id": "doc1", "text": "Python installation guide for beginners", "category": "tutorial", "difficulty": "beginner"},
    {"id": "doc2", "text": "PostgreSQL database query optimization", "category": "database", "difficulty": "advanced"},
    {"id": "doc3", "text": "Building REST API with FastAPI", "category": "tutorial", "difficulty": "intermediate"},