Tutorial ColPali: Visual Document Retrieval dengan Vision-Language Model
Visual document retrieval adalah salah satu tantangan terbesar dalam dunia AI saat ini. Bayangkan Anda memiliki ribuan dokumen PDF yang berisi tabel, grafik, diagram, dan layout kompleks. Metode tradisional menggunakan OCR dan text extraction sering kali gagal menangkap konteks visual yang krusial. ColPali hadir sebagai solusi revolusioner yang menggunakan vision-language model untuk melakukan retrieval langsung dari gambar dokumen tanpa perlu OCR.
Dalam tutorial ini, kita akan mempelajari cara menggunakan ColPali untuk membangun sistem document retrieval yang mampu memahami konten visual dokumen secara langsung.
Apa Itu ColPali?
ColPali adalah model retrieval yang menggabungkan arsitektur ColBERT (Contextualized Late Interaction over BERT) dengan PaliGemma (vision-language model dari Google). Alih-alih mengekstrak teks dari dokumen lalu melakukan retrieval pada teks tersebut, ColPali langsung memproses gambar halaman dokumen dan menghasilkan multi-vector embeddings yang merepresentasikan konten visual dan tekstual secara bersamaan.
Keunggulan utama ColPali:
- Tanpa OCR: Tidak perlu pipeline OCR yang rumit dan rawan error
- Memahami layout: Menangkap informasi dari tabel, grafik, dan diagram
- Multi-vector retrieval: Menggunakan late interaction seperti ColBERT untuk matching yang lebih presisi
- End-to-end: Satu model untuk seluruh pipeline retrieval
Instalasi
Persyaratan Sistem
- Python 3.10 atau lebih baru
- GPU dengan minimal 8GB VRAM (disarankan 16GB+)
- CUDA 11.8 atau lebih baru
Instalasi Library
pip install colpali-engine torch pillow pdf2image transformers
Untuk konversi PDF ke gambar, Anda juga memerlukan poppler:
# Ubuntu/Debian
sudo apt-get install poppler-utils
macOS
brew install poppler
Windows (via conda)
conda install -c conda-forge poppler
Instalasi Opsional
Jika Anda ingin menggunakan Qdrant sebagai vector store:
pip install qdrant-client
Basic Usage
Memuat Model ColPali
import torch
from colpaliengine.models import ColPali, ColPaliProcessor
modelname = "vidore/colpali-v1.3"
model = ColPali.frompretrained(
modelname,
torchdtype=torch.bfloat16,
devicemap="cuda"
)
processor = ColPaliProcessor.frompretrained(modelname)
Mengindeks Dokumen
Langkah pertama adalah mengkonversi halaman dokumen menjadi embeddings:
from PIL import Image
from pdf2image import convertfrompath
def indexpdf(pdfpath, model, processor):
"""Konversi PDF ke embeddings per halaman."""
pages = convertfrompath(pdfpath, dpi=144)
allembeddings = []
for i, page in enumerate(pages):
inputs = processor.processimages([page]).to(model.device)
with torch.nograd():
embeddings = model(inputs)
allembeddings.append({
"pagenum": i + 1,
"embeddings": embeddings.cpu(),
"image": page
})
print(f"Indexed page {i + 1}/{len(pages)}")
return allembeddings
Contoh penggunaan
docembeddings = indexpdf("laporankeuangan.pdf", model, processor)
print(f"Total halaman terindeks: {len(docembeddings)}")
Melakukan Query
Setelah dokumen terindeks, kita bisa melakukan pencarian:
def searchdocuments(query, docembeddings, model, processor, topk=3):
"""Cari halaman dokumen yang paling relevan dengan query."""
inputs = processor.processqueries([query]).to(model.device)
with torch.nograd():
queryembeddings = model(inputs)
scores = []
for doc in docembeddings:
score = processor.scoremultivector(
queryembeddings,
doc["embeddings"].to(model.device)
)
scores.append({
"pagenum": doc["pagenum"],
"score": score.item(),
"image": doc["image"]
})
scores.sort(key=lambda x: x["score"], reverse=True)
return scores[:topk]
Contoh pencarian
results = searchdocuments(
"Berapa total revenue Q3 2025?",
docembeddings,
model,
processor
)
for r in results:
print(f"Page {r['pagenum']}: Score {r['score']:.4f}")
Visualisasi Hasil
import matplotlib.pyplot as plt
def visualizeresults(query, results, maxdisplay=3):
"""Tampilkan hasil pencarian sebagai gambar."""
fig, axes = plt.subplots(1, min(len(results), maxdisplay), figsize=(18, 8))
if maxdisplay == 1:
axes = [axes]
fig.suptitle(f'Query: "{query}"', fontsize=14, fontweight="bold")
for idx, (ax, result) in enumerate(zip(axes, results[:maxdisplay])):
ax.imshow(result["image"])
ax.settitle(f"Page {result['pagenum']}\nScore: {result['score']:.4f}")
ax.axis("off")
plt.tightlayout()
plt.savefig("searchresults.png", dpi=150, bboxinches="tight")
plt.show()
visualizeresults("Total revenue Q3 2025", results)
Advanced Usage
Multi-Document Retrieval System
Untuk kasus nyata, Anda biasanya perlu mengindeks banyak dokumen sekaligus:
import os
import json
from pathlib import Path
class ColPaliRetriever:
def init(self, modelname="vidore/colpali-v1.3"):
self.model = ColPali.frompretrained(
modelname,
torchdtype=torch.bfloat16,
devicemap="cuda"
)
self.processor = ColPaliProcessor.frompretrained(modelname)
self.index = []
def adddocument(self, pdfpath, docid=None):
"""Tambahkan dokumen ke index."""
if docid is None:
docid = Path(pdfpath).stem
pages = convertfrompath(pdfpath, dpi=144)
for i, page in enumerate(pages):
inputs = self.processor.processimages([page]).to(self.model.device)
with torch.nograd():
embeddings = self.model(*inputs)
self.index.append({
"docid": docid,
"pagenum": i + 1,
"embeddings": embeddings.cpu(),
"source": str(pdfpath)
})
print(f"Added '{docid}': {len(pages)} pages")
def adddirectory(self, directory, pattern=".pdf"):
"""Index semua PDF dalam direktori."""
pdffiles = sorted(Path(directory).glob(pattern))
for pdf in pdffiles:
self.adddocument(str(pdf))
print(f"Total indexed: {len(self.index)} pages from {len(pdffiles)} documents")
def search(self, query, topk=5):
"""Cari halaman paling relevan dari seluruh index."""
inputs = self.processor.processqueries([query]).to(self.model.device)
with torch.nograd():
queryembeddings = self.model(inputs)
results = []
for entry in self.index:
score = self.processor.scoremultivector(
queryembeddings,
entry["embeddings"].to(self.model.device)
)
results.append({
"docid": entry["docid"],
"pagenum": entry["pagenum"],
"score": score.item(),
"source": entry["source"]
})
results.sort(key=lambda x: x["score"], reverse=True)
return results[:topk]
Penggunaan
retriever = ColPaliRetriever()
retriever.adddirectory("./documents/")
results = retriever.search("grafik pertumbuhan penjualan tahunan")
for r in results:
print(f"[{r['docid']}] Page {r['pagenum']}: {r['score']:.4f}")
Integrasi dengan Qdrant untuk Produksi
Untuk deployment produksi, simpan embeddings di vector database:
from qdrantclient import QdrantClient
from qdrantclient.models import (
Distance, VectorParams, PointStruct,
NamedVector, models
)
import numpy as np
import uuid
class ColPaliQdrantRetriever:
def init(self, modelname="vidore/colpali-v1.3", qdranturl="localhost", qdrantport=6333):
self.model = ColPali.frompretrained(
modelname,
torchdtype=torch.bfloat16,
devicemap="cuda"
)
self.processor = ColPaliProcessor.frompretrained(modelname)
self.client = QdrantClient(host=qdranturl, port=qdrantport)
self.collectionname = "colpalidocuments"
self.createcollection()
def createcollection(self):
"""Buat collection Qdrant jika belum ada."""
collections = [c.name for c in self.client.getcollections().collections]
if self.collectionname not in collections:
self.client.createcollection(
collectionname=self.collectionname,
vectorsconfig={
"colpali": VectorParams(
size=128,
distance=Distance.COSINE,
multivectorconfig=models.MultiVectorConfig(
comparator=models.MultiVectorComparator.MAXSIM
)
)
}
)
def indexdocument(self, pdfpath, metadata=None):
"""Index dokumen ke Qdrant."""
pages = convertfrompath(pdfpath, dpi=144)
points = []
for i, page in enumerate(pages):
inputs = self.processor.processimages([page]).to(self.model.device)
with torch.nograd():
embeddings = self.model(inputs)
emblist = embeddings.cpu().float().numpy().tolist()[0]
point = PointStruct(
id=str(uuid.uuid4()),
vector={"colpali": emblist},
payload={
"source": str(pdfpath),
"pagenum": i + 1,
"totalpages": len(pages),
(metadata or {})
}
)
points.append(point)
self.client.upsert(
collectionname=self.collectionname,
points=points
)
print(f"Indexed {len(points)} pages from {pdfpath}")
def search(self, query, topk=5):
"""Pencarian multi-vector di Qdrant."""
inputs = self.processor.processqueries([query]).to(self.model.device)
with torch.nograd():
queryembeddings = self.model(inputs)
querylist = queryembeddings.cpu().float().numpy().tolist()[0]
results = self.client.querypoints(
collectionname=self.collectionname,
query=querylist,
using="colpali",
limit=topk
)
return [
{
"score": r.score,
"source": r.payload["source"],
"pagenum": r.payload["pagenum"]
}
for r in results.points
]
Penggunaan
retriever = ColPaliQdrantRetriever()
retriever.indexdocument("annualreport.pdf", metadata={"type": "financial"})
results = retriever.search("revenue breakdown by region")
Batch Processing untuk Dataset Besar
Untuk mengindeks ribuan dokumen secara efisien:
from torch.utils.data import DataLoader, Dataset
class PDFPageDataset(Dataset):
def init(self, pdfpaths, dpi=144):
self.pages = []
for pdfpath in pdfpaths:
images = convertfrompath(pdfpath, dpi=dpi)
for i, img in enumerate(images):
self.pages.append({
"image": img,
"source": str(pdfpath),
"pagenum": i + 1
})
def len(self):
return len(self.pages)
def getitem(self, idx):
return self.pages[idx]
def batchindex(pdfpaths, model, processor, batchsize=4):
"""Index dokumen secara batch untuk efisiensi GPU."""
dataset = PDFPageDataset(pdfpaths)
allresults = []
for i in range(0, len(dataset), batchsize):
batch = [dataset[j] for j in range(i, min(i + batchsize, len(dataset)))]
images = [item["image"] for item in batch]
inputs = processor.processimages(images).to(model.device)
with torch.nograd():
embeddings = model(*inputs)
for j, item in enumerate(batch):
allresults.append({
"source": item["source"],
"pagenum": item["pagenum"],
"embeddings": embeddings[j:j+1].cpu()
})
print(f"Processed {min(i + batchsize, len(dataset))}/{len(dataset)} pages")
return allresults
ColPali + RAG Pipeline
Gabungkan ColPali dengan LLM untuk question answering:
import base64
from io import BytesIO
from openai import OpenAI
class VisualRAG:
def init(self, retriever):
self.retriever = retriever
self.llmclient = OpenAI()
def imagetobase64(self, image):
"""Konversi PIL Image ke base64."""
buffer = BytesIO()
image.save(buffer, format="PNG")
return base64.b64encode(buffer.getvalue()).decode()
def answer(self, question, topk=3):
"""Jawab pertanyaan berdasarkan dokumen yang di-retrieve."""
results = self.retriever.search(question, topk=topk)
imagecontents = []
for r in results:
pages = convertfrompath(r["source"], dpi=144,
firstpage=r["pagenum"],
lastpage=r["pagenum"])
if pages:
b64 = self.imagetobase64(pages[0])
imagecontents.append({
"type": "imageurl",
"imageurl": {
"url": f"data:image/png;base64,{b64}",
"detail": "high"
}
})
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": f"Based on the following document pages, answer this question: {question}"},
imagecontents
]
}
]
response = self.llmclient.chat.completions.create(
model="gpt-4o",
messages=messages,
maxtokens=1024
)
return {
"answer": response.choices[0].message.content,
"sources": [
{"source": r["source"], "page": r["pagenum"], "score": r["score"]}
for r in results
]
}
Penggunaan
rag = VisualRAG(retriever)
result = rag.answer("Berapa total aset perusahaan pada tahun 2025?")
print(result["answer"])
Best Practices
1. Optimasi Resolusi Gambar
Resolusi DPI sangat mempengaruhi kualitas retrieval. Gunakan 144 DPI sebagai default yang baik:
# Terlalu rendah - detail teks hilang
pageslow = convertfrompath(pdfpath, dpi=72)
Optimal - keseimbangan kualitas dan kecepatan
pagesoptimal = convertfrompath(pdfpath, dpi=144)
Terlalu tinggi - lambat tanpa peningkatan signifikan
pageshigh = convertfrompath(pdfpath, dpi=300)
2. Manajemen Memory GPU
ColPali menggunakan memory GPU yang cukup besar. Beberapa tips:
# Gunakan bfloat16 untuk mengurangi memory
model = ColPali.frompretrained(
modelname,
torchdtype=torch.bfloat16,
devicemap="cuda"
)
Bersihkan cache setelah batch processing
torch.cuda.emptycache()
Gunakan gradient checkpointing jika memory terbatas
model.gradientcheckpointingenable()
3. Preprocessing Dokumen
Tidak semua halaman perlu diindeks. Filter halaman yang tidak relevan:
def shouldindexpage(image, mintextratio=0.05):
"""Heuristic sederhana untuk menentukan apakah halaman perlu diindeks."""
import numpy as np
img
array = np.array(image.convert("L"))
nonwhite = np.sum(imgarray < 240)
total = imgarray.size
return (nonwhite / total) > mintextratio
Filter halaman kosong atau hampir kosong
pages = convertfrompath(pdfpath, dpi=144)
pagestoindex = [p for p in pages if shouldindexpage(p)]
4. Caching Embeddings
Simpan embeddings ke disk untuk menghindari re-computation:
import pickle
from pathlib import Path
def getorcreateembeddings(pdfpath, model, processor, cachedir="./cache"):
"""Load embeddings dari cache atau buat baru."""
cachepath = Path(cachedir) / f"{Path(pdfpath).stem}.pkl"
cachepath.parent.mkdir(existok=True)
if cachepath.exists():
with open(cachepath, "rb") as f:
return pickle.load(f)
embeddings = indexpdf(pdfpath, model, processor)
with open(cachepath, "wb") as f:
pickle.dump(embeddings, f)
return embeddings
5. Evaluasi Kualitas Retrieval
Ukur performa retrieval system Anda:
def evaluateretrieval(retriever, testqueries, groundtruth):
"""Evaluasi retrieval accuracy."""
metrics = {"mrr": 0, "recall
at5": 0, "recallat10": 0}
n = len(test
queries)
for query, expected in zip(testqueries, groundtruth):
results = retriever.search(query, topk=10)
retrievedpages = [(r["docid"], r["pagenum"]) for r in results]
for rank, page in enumerate(retrievedpages):
if page in expected:
metrics["mrr"] += 1 / (rank + 1)
break
top5 = set(retrievedpages[:5])
top10 = set(retrievedpages[:10])
metrics["recallat5"] += len(top5 & set(expected)) / len(expected)
metrics["recallat_10"] += len(top10 & set(expected)) / len(expected)
return {k: v / n for k, v in metrics.items()}
Perbandingan dengan Metode Tradisional
| Aspek | OCR + Text Retrieval | ColPali |
|-------|---------------------|---------|
| Setup | Kompleks (OCR pipeline) | Sederhana (1 model) |
| Tabel & Grafik | Sering gagal | Baik |
| Kecepatan Indexing | Cepat | Lebih lambat |
| Akurasi | Bergantung OCR | Konsisten tinggi |
| GPU Required | Tidak wajib | Ya |
| Multi-bahasa | Perlu konfigurasi OCR | Built-in |
Kesimpulan
ColPali membawa perubahan besar dalam cara kita melakukan document retrieval. Dengan menghilangkan kebutuhan OCR dan langsung memproses visual dokumen, ColPali mampu menangkap informasi yang sering terlewat oleh metode tradisional seperti tabel, grafik, dan layout kompleks.
Poin-poin penting dari tutorial ini:
Untuk project produksi, pastikan Anda menggunakan vector database seperti Qdrant atau Milvus untuk menyimpan embeddings, implementasikan caching yang baik, dan lakukan evaluasi berkala terhadap kualitas retrieval system Anda.