Complete Pinecone Tutorial: Vector Database for AI and Semantic Search

Pinecone is a managed vector database designed specifically for storing, indexing, and performing similarity searches on high-dimensional vector data. In the modern AI ecosystem, Pinecone has become one of the top choices for building Retrieval-Augmented Generation (RAG) systems, semantic search, recommendation engines, and various embedding-based applications.

Unlike traditional databases that rely on exact matching, Pinecone enables search based on semantic similarity. When you convert text, images, or other data into vector embeddings, Pinecone can find the most similar items in milliseconds, even at the scale of billions of vectors.

In this tutorial, we will learn how to use Pinecone from installation to advanced usage, including integration with popular embedding models and building a complete RAG pipeline.

Why Choose Pinecone?

Before diving into implementation, here are several reasons why Pinecone is a popular choice for vector databases:

Fully Managed: No need to manage infrastructure, scaling, or database maintenance

High Performance: Similarity search in milliseconds at massive scale

Serverless Architecture: Usage-based pricing model suitable for projects of any size

Metadata Filtering: Combine vector search with metadata filters for more precise results

Namespace Support: Isolate data within a single index using namespaces

Hybrid Search: Support for sparse-dense vector search

Integration Ecosystem: Compatible with LangChain, LlamaIndex, Haystack, and other AI frameworks

Installation and Setup

Prerequisites

Make sure you have Python 3.8 or newer installed on your system.

Installing the Pinecone Client

pip install pinecone-client

For installation with additional useful dependencies:

pip install pinecone-client sentence-transformers openai python-dotenv

Creating an Account and API Key

Sign up for a free account at pinecone.io

After logging in, open the dashboard and copy your API Key

Store the API Key securely, for example in a .env file

# .env
PINECONEAPIKEY=your-api-key-here

Initializing the Client

import os
from dotenv import loaddotenv

from pinecone import Pinecone

loaddotenv()

pc = Pinecone(apikey=os.getenv("PINECONEAPIKEY"))


Verify connection
print("Available indexes:", pc.listindexes().names())

Core Concepts

Index

An index is the primary storage unit in Pinecone, similar to a "table" in a relational database. Each index is configured with a specific vector dimension and similarity metric.

Namespace

Namespaces allow you to partition data within a single index. Each namespace is independent, so queries on one namespace will not return results from another.

Vectors and Metadata

Each record in Pinecone consists of:

• ID: A unique identifier for the vector
• Values: A numerical array (the vector embedding)
• Metadata: Additional key-value pair data that can be filtered

Similarity Metrics

Pinecone supports three metrics:

• Cosine: Measures the angle between two vectors (most common for text)
• Euclidean: Measures the geometric distance between two points
• Dot Product: Measures the projection of one vector onto another

Basic Usage

Creating an Index

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(apikey=os.getenv("PINECONEAPIKEY"))


indexname = "tutorial-index"

if indexname not in pc.listindexes().names():
    pc.createindex(

        name=indexname,
        dimension=384,  # Match your embedding model
        metric="cosine",
        spec=ServerlessSpec(
            cloud="aws",
            region="us-east-1"
        )
    )
    print(f"Index '{indexname}' created successfully")


Connect to the index
index = pc.Index(indexname)

View index statistics
print(index.describeindexstats())

Storing Vectors (Upsert)

# Simple vector upsert
vectors = [
    {
        "id": "doc-1",
        "values": [0.1, 0.2, 0.3, ...],  # 384 dimensions
        "metadata": {
            "title": "Introduction to Machine Learning",
            "category": "tutorial",
            "language": "en"
        }
    },
    {
        "id": "doc-2",
        "values": [0.4, 0.5, 0.6, ...],
        "metadata": {
            "title": "Deep Learning Fundamentals",
            "category": "article",
            "language": "en"
        }
    }
]

index.upsert(vectors=vectors)

Upsert with an Embedding Model

In practice, you will use an embedding model to generate vectors:

from sentencetransformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

documents = [
    {"id": "doc-1", "text": "Machine learning is a branch of artificial intelligence", "category": "ai"},
    {"id": "doc-2", "text": "Python is a popular programming language for data science", "category": "programming"},
    {"id": "doc-3", "text": "Neural networks are inspired by how the human brain works", "category": "ai"},
    {"id": "doc-4", "text": "Relational databases use tables to store data", "category": "database"},
    {"id": "doc-5", "text": "Natural Language Processing processes and understands human language", "category": "ai"},
]

Generate embeddings
texts = [doc["text"] for doc in documents]
embeddings = model.encode(texts)

Prepare data for upsert
vectorstoupsert = []

for doc, embedding in zip(documents, embeddings):
    vectorstoupsert.append({

        "id": doc["id"],
        "values": embedding.tolist(),
        "metadata": {
            "text": doc["text"],
            "category": doc["category"]
        }
    })

Upsert to Pinecone
index.upsert(vectors=vectorstoupsert)

print(f"Successfully stored {len(vectorstoupsert)} vectors")

Performing Queries (Similarity Search)

# Query using text
querytext = "How does artificial intelligence work?"
queryembedding = model.encode(querytext).tolist()

results = index.query(
    vector=queryembedding,

    topk=3,
    includemetadata=True

)

print(f"Query: {querytext}\n")
for match in results["matches"]:
    print(f"Score: {match['score']:.4f}")
    print(f"Text: {match['metadata']['text']}")
    print(f"Category: {match['metadata']['category']}")
    print()

Output:

Query: How does artificial intelligence work?

Score: 0.7823
Text: Machine learning is a branch of artificial intelligence
Category: ai

Score: 0.6541
Text: Neural networks are inspired by how the human brain works
Category: ai

Score: 0.6102
Text: Natural Language Processing processes and understands human language
Category: ai

Query with Metadata Filters

# Search only within a specific category
results = index.query(
    vector=queryembedding,

    topk=3,
    includemetadata=True,

    filter={
        "category": {"$eq": "programming"}
    }
)

Filter with complex conditions
results = index.query(
    vector=queryembedding,
    topk=5,

    includemetadata=True,
    filter={
        "$and": [
            {"category": {"$in": ["ai", "programming"]}},
            {"language": {"$eq": "en"}}
        ]
    }
)

Other CRUD Operations

# Fetch vectors by ID
fetched = index.fetch(ids=["doc-1", "doc-2"])
print(fetched)

Update vector metadata
index.update(
    id="doc-1",
    setmetadata={"category": "machine-learning", "updated": True}

)

Delete vectors by ID
index.delete(ids=["doc-4"])

Delete by metadata filter
index.delete(filter={"category": {"$eq": "deprecated"}})

Delete all vectors in a namespace
index.delete(deleteall=True, namespace="old-data")

Advanced Usage

Batch Upsert for Large Datasets

When working with large datasets, upserts should be performed in batches:

import itertools

def chunks(iterable, batchsize=100):

    it = iter(iterable)
    chunk = list(itertools.islice(it, batchsize))
    while chunk:
        yield chunk
        chunk = list(itertools.islice(it, batchsize))


Suppose you have thousands of documents
largedataset = [...]  # List of documents

allvectors = []

for i, doc in enumerate(largedataset):
    embedding = model.encode(doc["text"]).tolist()
    allvectors.append({

        "id": f"doc-{i}",
        "values": embedding,
        "metadata": {"text": doc["text"], "source": doc["source"]}
    })

Upsert in batches
for batch in chunks(allvectors, batchsize=100):

    index.upsert(vectors=batch)

print(f"Total vectors upserted: {len(allvectors)}")

Using Namespaces

# Store data in different namespaces
index.upsert(
    vectors=productvectors,

    namespace="products"
)

index.upsert(
    vectors=articlevectors,
    namespace="articles"
)

Query a specific namespace
results = index.query(
    vector=queryembedding,

    topk=5,
    namespace="products",
    includemetadata=True

)

Statistics per namespace
stats = index.describeindexstats()

print(stats)
{'dimension': 384, 'namespaces': {'products': {'vectorcount': 1000}, 'articles': {'vectorcount': 500}}}

Building a RAG Pipeline with Pinecone

Here is a complete example of building a RAG system using Pinecone and OpenAI:

import os
from pinecone import Pinecone, ServerlessSpec
from openai import OpenAI
from sentencetransformers import SentenceTransformer

Initialize
pc = Pinecone(apikey=os.getenv("PINECONEAPIKEY"))

openaiclient = OpenAI(apikey=os.getenv("OPENAIAPIKEY"))

embedmodel = SentenceTransformer("all-MiniLM-L6-v2")

INDEXNAME = "rag-knowledge-base"

NAMESPACE = "documents"

Create index if it doesn't exist
if INDEXNAME not in pc.listindexes().names():

    pc.createindex(
        name=INDEXNAME,

        dimension=384,
        metric="cosine",
        spec=ServerlessSpec(cloud="aws", region="us-east-1")
    )

index = pc.Index(INDEXNAME)

def ingestdocuments(documents):

    """Ingest documents into Pinecone"""
    vectors = []
    for i, doc in enumerate(documents):
        # Simple chunking
        textchunks = splittext(doc["content"], chunksize=500, overlap=50)

        for j, chunk in enumerate(textchunks):

            embedding = embedmodel.encode(chunk).tolist()
            vectors.append({
                "id": f"{doc['id']}-chunk-{j}",
                "values": embedding,
                "metadata": {
                    "text": chunk,
                    "source": doc["source"],
                    "title": doc["title"]
                }
            })

    # Batch upsert
    for k in range(0, len(vectors), 100):
        batch = vectors[k:k+100]
        index.upsert(vectors=batch, namespace=NAMESPACE)

    return len(vectors)

def splittext(text, chunksize=500, overlap=50):
    """Split text into chunks with overlap"""
    words = text.split()
    chunks = []
    for i in range(0, len(words), chunksize - overlap):

        chunk = " ".join(words[i:i + chunksize])
        if chunk:
            chunks.append(chunk)
    return chunks

def retrievecontext(query, topk=5):
    """Retrieve relevant context from Pinecone"""
    queryembedding = embedmodel.encode(query).tolist()

    results = index.query(
        vector=queryembedding,

        topk=topk,

        includemetadata=True,
        namespace=NAMESPACE
    )

    contexts = []
    for match in results["matches"]:
        if match["score"] > 0.3:  # Minimum threshold
            contexts.append({
                "text": match["metadata"]["text"],
                "source": match["metadata"]["source"],
                "score": match["score"]
            })

    return contexts

def generateanswer(query, contexts):

    """Generate an answer using an LLM with context from Pinecone"""
    contexttext = "\n\n".join([
        f"[Source: {ctx['source']}]\n{ctx['text']}" 
        for ctx in contexts
    ])

    response = openaiclient.chat.completions.create(

        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a helpful assistant that answers questions based on the provided context. "
                    "Only answer based on information in the context. "
                    "If the information is not available in the context, say you couldn't find it."
                )
            },
            {
                "role": "user",
                "content": f"Context:\n{contexttext}\n\nQuestion: {query}"
            }
        ],
        temperature=0.1
    )

    return response.choices[0].message.content

def ragquery(query):

    """Complete RAG pipeline"""
    print(f"Query: {query}")

    # Retrieve
    contexts = retrievecontext(query)
    print(f"Found {len(contexts)} relevant contexts")

    # Generate
    answer = generateanswer(query, contexts)

    print(f"\nAnswer: {answer}")

    # Include sources
    print("\nSources:")
    for ctx in contexts:
        print(f"  - {ctx['source']} (score: {ctx['score']:.4f})")

    return answer

Example usage
knowledgebase = [
    {
        "id": "kb-1",
        "title": "Python Guide",
        "source": "docs/python-guide.md",
        "content": "Python is a high-level programming language that is easy to learn..."
    },
    {
        "id": "kb-2",
        "title": "Machine Learning Basics",
        "source": "docs/ml-basics.md",
        "content": "Machine learning is a subset of artificial intelligence..."
    }
]

Ingest
count = ingestdocuments(knowledgebase)
print(f"Successfully ingested {count} chunks")

Query
answer = ragquery("What is machine learning?")

Hybrid Search (Sparse-Dense)

Pinecone supports hybrid search that combines semantic search (dense vectors) with keyword search (sparse vectors):

from pineconetext.sparse import BM25Encoder

Initialize BM25 encoder for sparse vectors
bm25 = BM25Encoder()

Fit BM25 on corpus
corpus = [doc["text"] for doc in documents]
bm25.fit(corpus)

Generate sparse and dense vectors
query = "machine learning tutorial python"

Dense embedding
densevector = embedmodel.encode(query).tolist()

Sparse embedding (BM25)
sparsevector = bm25.encodequeries(query)

Hybrid query
results = index.query(
    vector=densevector,

    sparsevector=sparsevector,

    topk=10,
    includemetadata=True,

    alpha=0.5  # Balance between dense and sparse (0 = sparse only, 1 = dense only)
)

Integration with LangChain

from langchainpinecone import PineconeVectorStore
from langchaincommunity.embeddings import HuggingFaceEmbeddings

from langchain.textsplitter import RecursiveCharacterTextSplitter
from langchaincommunity.documentloaders import TextLoader

Load documents
loader = TextLoader("knowledgebase.txt")

documents = loader.load()

Split documents
textsplitter = RecursiveCharacterTextSplitter(
    chunksize=1000,

    chunkoverlap=200
)
docs = textsplitter.splitdocuments(documents)

Embedding model
embeddings = HuggingFaceEmbeddings(
    modelname="all-MiniLM-L6-v2"

)

Create vector store
vectorstore = PineconeVectorStore.fromdocuments(
    documents=docs,
    embedding=embeddings,
    indexname="langchain-index",

    namespace="docs"
)

Similarity search
results = vectorstore.similaritysearch(
    "What is a neural network?",
    k=3
)

for doc in results:
    print(doc.pagecontent)

    print()

Use as a retriever
retriever = vectorstore.asretriever(
    searchtype="similarity",

    searchkwargs={"k": 5}
)

Integration with LlamaIndex

from llamaindex.core import VectorStoreIndex, SimpleDirectoryReader
from llamaindex.vectorstores.pinecone import PineconeVectorStore

from pinecone import Pinecone

pc = Pinecone(apikey=os.getenv("PINECONEAPIKEY"))
pineconeindex = pc.Index("llamaindex-demo")


Create vector store
vectorstore = PineconeVectorStore(pineconeindex=pineconeindex)

Load and index documents
documents = SimpleDirectoryReader("./data").loaddata()

index = VectorStoreIndex.fromdocuments(
    documents,
    vectorstore=vectorstore
)

Query
queryengine = index.asqueryengine()

response = queryengine.query("Explain the concept of transfer learning")
print(response)

Best Practices

1. Choose the Right Embedding Dimensions

The embedding dimension must match the model you are using:

| Model | Dimensions | Use Case |

|-------|-----------|----------|

| all-MiniLM-L6-v2 | 384 | General purpose, lightweight |

| text-embedding-3-small | 1536 | OpenAI, balanced performance-cost |

| text-embedding-3-large | 3072 | OpenAI, maximum accuracy |

| BAAI/bge-large-en-v1.5 | 1024 | Open source, high performance |

2. Effective Chunking Strategies

from langchain.textsplitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunksize=512,
    chunkoverlap=50,

    separators=["\n\n", "\n", ". ", " ", ""]
)

Chunks that are too small lose context
Chunks that are too large dilute relevance
256-1024 tokens is a good range for most cases

3. Leverage Metadata Filtering

# Store useful metadata for filtering
vector = {
    "id": "article-123",
    "values": embedding,
    "metadata": {
        "text": chunktext,
        "source": "blog",
        "date": "2026-01-15",
        "language": "en",
        "category": "tutorial",
        "author": "Ruby Abdullah"
    }
}

Combine vector search with metadata filters
results = index.query(
    vector=queryembedding,

    topk=10,
    filter={
        "$and": [
            {"category": {"$eq": "tutorial"}},
            {"language": {"$eq": "en"}}
        ]
    },
    includemetadata=True

)

4. Monitoring and Observability

# Check index statistics periodically
stats = index.describeindexstats()

print(f"Total vectors: {stats['totalvectorcount']}")

print(f"Dimensions: {stats['dimension']}")
for ns, nsstats in stats.get("namespaces", {}).items():
    print(f"Namespace '{ns}': {nsstats['vectorcount']} vectors")

5. Error Handling and Retry

import time
from pinecone.exceptions import PineconeException

def safeupsert(index, vectors, namespace="", maxretries=3):
    for attempt in range(maxretries):

        try:
            index.upsert(vectors=vectors, namespace=namespace)
            return True
        except PineconeException as e:
            if attempt < maxretries - 1:
                waittime = 2  attempt

                print(f"Retry {attempt + 1}/{maxretries} after {waittime}s: {e}")

                time.sleep(waittime)
            else:
                raise
    return False

6. Cost Optimization

• Use serverless indexes for variable workloads
• Choose embedding dimensions appropriate for your needs (not always the largest)
• Leverage namespaces instead of creating many indexes
• Regularly delete data that is no longer relevant
• Use metadata filters to narrow the search space before vector similarity computation

Common Mistakes to Avoid

Dimension mismatch: Ensure the index dimension matches the embedding model output

Not storing original text: Always save the original text in metadata so it can be displayed

Chunks too large: Chunks over 1000 tokens reduce retrieval precision

Ignoring score thresholds: Always filter results with a minimum score

Upsert without batching: For large data, always use batch upserts

Not using namespaces**: Mixing different data types without isolation

Conclusion

Pinecone provides a robust and easy-to-use vector database solution for various AI applications. From simple semantic search to complex RAG pipelines, Pinecone offers high performance without the complexity of managing infrastructure.

Key takeaways:

• Choose the embedding model and dimensions that match your requirements
• Leverage metadata filtering to narrow search results
• Use batch processing for operations on large datasets
• Apply namespaces to organize your data
• Monitor usage and optimize costs regularly

With the foundational and advanced techniques covered in this tutorial, you are now ready to build AI applications that effectively leverage vector search using Pinecone.