Vespa: The Big Data Serving Engine for Vector Search, Ranking, and Recommendation at Scale

Hey friends, in this tutorial I want to introduce you to one of those tools that in my opinion often gets overlooked even though it is incredibly powerful, and it is called Vespa. If you are building an application that needs search, recommendation, or ranking that has to run fast across millions or even billions of documents, Vespa is going to be a very useful companion. I have personally been playing with Vespa a lot lately for hybrid search needs, and honestly the results have amazed me. So in this article I am going to explain everything from the core concepts all the way to actually deploying an application using Python, so you can practice on your own right away.

Introduction

Before we jump into installation, I want to tell you what Vespa actually is. Vespa is an open-source big data serving engine, originally developed by Yahoo and now maintained by Vespa.ai. What makes it different from a regular database or a regular search engine is that it is designed specifically to serve real-time queries over huge amounts of data, while performing heavy computation like machine learning ranking, along with vector search for semantic matching. So imagine this friends, in one single system you can combine traditional text search using BM25, plus vector search using embeddings, then rank everything using an ML model you own. All of that runs in a matter of milliseconds.

So you do not get lost, let me explain a few core concepts in Vespa that you absolutely need to understand from the start:

Documents

In Vespa, the most basic unit of data is called a document. A single document is like a row in a database, for example one news article, one product, or one person profile. Every document has a unique ID and contains a collection of fields. Vespa stores these documents and can update, delete, or query them in real time without having to reindex everything from scratch.

Schema

The schema is like a blueprint that defines the shape of your documents. Inside the schema you specify what fields exist, their data types, which fields get indexed for search, and how to rank the results. A single Vespa application can have one or more schemas. So the schema is very important because it determines the search and ranking capabilities of your application.

Fields

A field is an attribute inside a document. For example an article has a title field, a body field, and an embedding field. Every field has a type, for example string for text, int for numbers, array for lists, and tensor for vectors. What is interesting is that you can configure attributes per field, such as whether it is indexed (for text search), whether it is stored as an attribute (for fast filtering and sorting), and whether it uses summary (to be displayed in the results).

Rank Profiles

This is my favorite part. A rank profile defines how Vespa scores each document that matches a query. You can write ranking expressions using Vespa built-in features like bm25, closeness (for vector similarity), or even call machine learning models like ONNX. You can also build multi-phase ranking, for example the first phase uses cheap computation to filter candidates, then the second phase uses a more expensive model to rerank the best candidates.

Hybrid Search: BM25 plus Vector (ANN)

The concept I use the most is hybrid search. So here is the thing friends, traditional text search using BM25 is great at finding exact keyword matches, but it does not understand meaning. Meanwhile vector search using embeddings understands semantic meaning, but sometimes it misses on specific keywords like product names or codes. So hybrid search combines both, taking the best of both worlds. For vector search, Vespa uses an algorithm called ANN, short for Approximate Nearest Neighbor, via the HNSW structure, so that vector search stays fast even when the data is in the millions.

Alright, now that you understand the basic concepts, let us start practicing.

Installation

To start using Vespa, the easiest way in my opinion is to use Docker to run Vespa itself, then use the Python library pyvespa to define and deploy the application. I am going to explain both.

Running Vespa via Docker

First make sure Docker is installed on your machine. If not, install Docker Desktop or Docker Engine according to your OS first. After that, we can immediately pull and run the official Vespa image. Vespa needs a fair amount of memory, so make sure your Docker is allocated at least 4GB of RAM, ideally 8GB.

docker run --detach --name vespa --hostname vespa-container \
  --publish 8080:8080 --publish 19071:19071 \
  vespaengine/vespa

Let me explain the ports. Port 8080 is for querying and feeding documents, so this is the one we will use most often. Port 19071 is for the config server, whose job is to receive our application deployments. To make sure the container is running, you can check its status:

docker exec vespa vespa-logfmt -l warning,error

If you want to wait until the config server is truly ready before deploying, you can poll the config server health endpoint:

curl -s --head http://localhost:19071/state/v1/health

If the response is 200 OK, then the config server is ready to receive deployments.

Install the Vespa CLI

Vespa has a CLI that is really nice for quick operations like deploy, feed, and query from the terminal. To install on macOS using Homebrew:

brew install vespa-cli

For other operating systems, you can download the binary from the Vespa GitHub releases page. Once installed, you can point the CLI at your local instance:

vespa config set target local
vespa status deploy --wait 300

This CLI is useful later for debugging and manual operations. But for our main workflow in this tutorial, I am going to focus on using Python.

Install pyvespa

Now we install the Python library. I recommend using a virtual environment to keep things clean.

python3 -m venv venv
source venv/bin/activate
pip install pyvespa

This pyvespa library is cool because we can define the schema, rank profiles, and the application programmatically using Python, then deploy directly to a Vespa instance. So we do not need to write XML files and schemas manually, although you still can if you want to.

Basic Usage

Alright now the fun part. Let us build our first Vespa application using pyvespa. The scenario is that we build a search engine for articles, where each article has a title, a body, and an embedding for semantic search.

Defining the Application Package

In pyvespa, we start by creating something called an ApplicationPackage. This is like a container that gathers all the schemas and configuration of our application.

from vespa.package import (
    ApplicationPackage,
    Schema,
    Document,
    Field,
    FieldSet,
    RankProfile,
)

apppackage = ApplicationPackage(name="article")


apppackage.schema.addfields(

    Field(name="docid", type="string", indexing=["attribute", "summary"]),
    Field(
        name="title",
        type="string",
        indexing=["index", "summary"],
        index="enable-bm25",
    ),
    Field(
        name="body",
        type="string",
        indexing=["index", "summary"],
        index="enable-bm25",
    ),
)

Take a look friends, each Field has an indexing parameter whose value is a list of instructions. If I pass index, it means the field is indexed for text search. If attribute, it means it is stored in memory for fast filtering and sorting. If summary, it means the field will be included in the query results. Then for title and body I add index="enable-bm25" so that we can use the BM25 ranking feature on those fields later.

Adding a FieldSet

A FieldSet is like a group of fields we can search all at once. So instead of searching only in the title or only in the body, we can create a fieldset that combines both.

apppackage.schema.addfieldset(
    FieldSet(name="default", fields=["title", "body"])
)

apppackage.schema.addrankprofile(
    RankProfile(
        name="bm25",
        firstphase="bm25(title) + bm25(body)",

    )
)

from vespa.deployment import VespaDocker

vespadocker = VespaDocker(port=8080)
app = vespadocker.deploy(applicationpackage=apppackage)


print("Application deployed successfully and ready to use")

documents = [
    {
        "docid": "1",
        "title": "A Guide to Learning Machine Learning",
        "body": "Machine learning is a branch of AI that learns from data.",
    },
    {
        "docid": "2",

        "title": "Special Fried Rice Recipe",
        "body": "Tasty fried rice needs cold rice and high heat.",
    },
    {
        "docid": "3",
        "title": "Deep Learning and Neural Network Basics",
        "body": "Neural networks are inspired by how the human brain works.",
    },
]

for d in documents:
    response = app.feeddatapoint(
        schema="article",
        dataid=d["docid"],
        fields=d,
    )
    print(response.statuscode)

def datagenerator(documents):
    for d in documents:
        yield {"id": d["docid"], "fields": d}

app.feediterable(

    iter=datagenerator(documents),
    schema="article",
    callback=lambda response, id: print(f"Feed {id}: {response.statuscode}"),

)

with app.syncio() as session:
    response = session.query(
        yql="select  from article where userQuery()",
        query="learning machine learning",
        ranking="bm25",
    )

for hit in response.hits:
    print(hit["fields"]["title"], "->", hit["relevance"])

from vespa.package import HNSW

apppackage.schema.addfields(

    Field(
        name="embedding",
        type="tensor(x[384])",
        indexing=["attribute", "index"],
        ann=HNSW(
            distancemetric="angular",
            maxlinkspernode=16,

            neighborstoexploreatinsert=200,

        ),
    )
)

Notice the field type tensor(x[384]), this means a float vector of dimension 384 with a dimension named x. Then I attach an HNSW index with the angular distance metric, which is suitable for cosine similarity. HNSW is what makes ANN search fast even when the data is large. The maxlinkspernode and neighborstoexploreatinsert parameters control the trade-off between speed, accuracy, and memory usage.

Hybrid Rank Profile

Now we build a rank profile that combines BM25 with vector similarity. We use the closeness function to measure the closeness of the query vector to the document vector.

from vespa.package import RankProfile, Function

apppackage.schema.addrankprofile(

    RankProfile(
        name="hybrid",
        inputs=[("query(qembedding)", "tensor(x[384])")],
        functions=[
            Function(
                name="bm25score",

                expression="bm25(title) + bm25(body)",
            ),
            Function(
                name="vectorscore",
                expression="closeness(field, embedding)",
            ),
        ],
        firstphase="bm25score + 100  vectorscore",

    )
)

from sentencetransformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

documents = [
    {"docid": "1", "title": "Machine Learning Guide",
     "body": "Machine learning learns from historical data."},
    {"docid": "2", "title": "Fried Rice Recipe",

     "body": "Fried rice needs cold rice and complete seasoning."},
]

def datagenerator(docs):
    for d in docs:
        text = d["title"] + ". " + d["body"]
        vector = model.encode(text).tolist()
        fields = dict(d)
        fields["embedding"] = vector
        yield {"id": d["docid"], "fields": fields}


app.feediterable(
    iter=datagenerator(documents),

    schema="article",
    callback=lambda response, id: print(id, response.statuscode),
)

querytext = "how to learn artificial intelligence"
qvector = model.encode(querytext).tolist()


with app.syncio() as session:
    response = session.query(
        yql=(
            "select  from article where "
            "userQuery() or "
            "({targetHits:100}nearestNeighbor(embedding, qembedding))"
        ),
        query=querytext,

        ranking="hybrid",
        body={"input.query(qembedding)": qvector},

    )

for hit in response.hits:
    print(hit["fields"]["title"], "->", round(hit["relevance"], 4))

Take a look at the YQL. I use userQuery() for the BM25 text part, then OR it with nearestNeighbor(embedding, qembedding) for the vector part. The {targetHits:100} annotation tells Vespa to grab roughly 100 nearest candidates via HNSW. Since I use OR, documents that match on either path (text or vector) will enter the candidate set, then get ranked together using the hybrid rank profile. This is the essence of hybrid search in Vespa.

Multi-Phase Ranking

For large production applications, we often use two-phase ranking to be efficient. The first phase uses cheap computation to filter, the second phase uses expensive computation to rerank the best candidates.

apppackage.schema.addrankprofile(
    RankProfile(
        name="hybridtwophase",

        inputs=[("query(qembedding)", "tensor(x[384])")],
        firstphase="bm25(title) + bm25(body)",

        secondphase={
            "expression": "closeness(field, embedding)",
            "rerank-count": 50,
        },
    )
)

Here the first phase filters using cheap BM25, then rerank-count: 50 tells Vespa to only rerank the top 50 documents using vector closeness in the second phase. This is a very compute-efficient strategy for large scale data. You do not need to compute vector similarity for all documents, only for the best candidates from the first phase.

Filtering and Pagination

Vespa is also great at filtering. Because some of our fields are stored as attributes, we can filter quickly. For example if we have a category field:

with app.syncio() as session:
    response = session.query(
        yql=(
            "select * from article where userQuery() "
            "and category contains 'technology'"
        ),
        query="machine learning",
        ranking="bm25",
        hits=10,
        offset=0,
    )

The hits parameter controls how many results per page, and offset is for pagination. Filtering using attributes is very fast because they are stored in memory.

Best Practices

After spending quite a while playing with Vespa, there are a few things I want to share so you do not fall into the same holes I did back then.

Choose Indexing Attributes Wisely

Do not just assign every attribute to every field friends. Every attribute is stored in memory, so if you assign an attribute to a body field full of long text, Vespa memory will be extremely wasteful. Give index to fields whose text you want to search, attribute only to fields you want to filter or sort on, and summary to fields you want to display in the results. Saving memory is the key to performance in Vespa.

Tune Hybrid Search Weights

Like I said earlier, combining BM25 scores with vector similarity requires tuning. The scales of these two scores are very different, so you have to experiment with the weights. A good approach is to normalize each score first before combining. Vespa has normalization features like normalizelinear that can make score combination more fair. I recommend building a small evaluation dataset to measure ranking quality before and after tuning.

HNSW parameters like maxlinkspernode and neighborstoexploreatinsert control the trade-off. Higher values increase ANN accuracy but also increase memory and insert time. To start, the default values are usually good enough. If you need higher recall, increase targetHits in the query and neighborstoexploreatinsert in the schema gradually while measuring the results.

Use Efficient Feeding

To feed a lot of data, do not use a feeddatapoint loop one by one because it is slow. Use feed_iterable which feeds in parallel. If your data is in the millions, consider using the dedicated Vespa feed client which is designed for high throughput. Also do not forget to monitor the response status in the callback so you know if any documents failed to get in.

Leverage Multi-Phase Ranking

For production, I almost always use multi-phase ranking. Put cheap computation in the first phase to narrow hundreds of thousands of candidates down to tens or hundreds, then put expensive computation like ML models or cross-encoders in the second phase. This keeps latency low without sacrificing ranking quality. The principle is: filter coarsely first with something cheap, then polish finely with something expensive on the best candidates.

Test in a Production-Like Environment

Vespa has many configuration knobs regarding memory, number of nodes, and resource allocation. The behavior on your laptop can be very different from a production cluster. So before release, test first with data volume and query patterns similar to production. Pay attention to metrics like query latency, memory usage, and recall from the ANN search.

Back Up Your Schema in Version Control

Because you use pyvespa, all your application definitions live in Python code. Take advantage of this by storing your schema and rank profile code in git. So every ranking change can be reviewed, rolled back, and tracked. This is a practice that is often forgotten but very important for team collaboration.

Conclusion

Alright friends, that was our journey from zero to being able to build an advanced search application using Vespa. We covered the core concepts starting from documents, schemas, fields, all the way to rank profiles. We also practiced installing via Docker and the Vespa CLI, then built an application using pyvespa, fed documents, and queried using BM25. What I found most exciting, we already tried hybrid search that combines the power of BM25 and vector search via ANN, plus efficient multi-phase ranking for large scale.

What I love about Vespa is that it is not just a vector database or an ordinary search engine. It is truly a complete serving engine that can handle text search, vector search, filtering, and ML-based ranking in one integrated system. For those of you building search or recommendation applications that need scale and flexibility, I highly recommend you try Vespa.

My advice, start small first. Build a simple application like the example above, play with the rank profiles, then slowly add embeddings and hybrid search. Once you are comfortable, then explore advanced features like ONNX models for ranking, more complex tensor operations, and multi-node deployment for production. The official Vespa documentation is also very complete, so do not hesitate to read around there.

Okay that is it from me for now. I hope this tutorial is useful and gets you more excited to dig into Vespa. If you get confused or want to discuss, do not hesitate to ask. Happy coding and see you in the next tutorial friends. Keep up the spirit of learning technology.