Tutorial Presidio: Deteksi dan Anonimisasi Data Pribadi (PII) dengan Python
Dalam era AI dan big data, perlindungan data pribadi bukan lagi pilihan melainkan keharusan. Regulasi seperti UU PDP (Undang-Undang Pelindungan Data Pribadi) di Indonesia, GDPR di Eropa, dan berbagai regulasi global lainnya mengharuskan organisasi untuk mengelola data pribadi secara bertanggung jawab. Microsoft Presidio hadir sebagai solusi open-source yang powerful untuk mendeteksi, menganalisis, dan menganonimisasi PII (Personally Identifiable Information) dalam teks dan data terstruktur.
Tutorial ini akan memandu Anda langkah demi langkah mulai dari instalasi hingga penggunaan tingkat lanjut, lengkap dengan contoh kode yang bisa langsung dipraktekkan di project Anda.
Apa Itu Microsoft Presidio?
Presidio adalah SDK open-source dari Microsoft yang menyediakan modul untuk deteksi dan anonimisasi data sensitif. Library ini mendukung berbagai jenis PII seperti nama, email, nomor telepon, nomor KTP, kartu kredit, dan masih banyak lagi. Presidio terdiri dari dua komponen utama:
- Presidio Analyzer: Mendeteksi dan mengklasifikasikan entitas PII dalam teks
- Presidio Anonymizer: Menganonimisasi entitas PII yang terdeteksi dengan berbagai strategi
Presidio menggunakan kombinasi Named Entity Recognition (NER) berbasis SpaCy/Transformers, regular expressions, dan rule-based logic untuk mendeteksi PII dengan akurasi tinggi.
Instalasi
Instalasi Dasar
pip install presidio-analyzer presidio-anonymizer
Instalasi dengan SpaCy Model
Presidio membutuhkan model NLP untuk deteksi entitas. Untuk bahasa Inggris:
python -m spacy download encoreweblg
Untuk dukungan multi-bahasa (termasuk Indonesia):
python -m spacy download xxentwikism
Instalasi Lengkap (dengan Transformers)
Jika Anda ingin menggunakan model Transformers untuk akurasi yang lebih tinggi:
pip install presidio-analyzer presidio-anonymizer torch transformers
python -m spacy download encoreweblg
Verifikasi Instalasi
from presidioanalyzer import AnalyzerEngine
from presidioanonymizer import AnonymizerEngine
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
print("Presidio berhasil diinstal!")
Basic Usage: Deteksi PII
Mendeteksi PII dalam Teks
from presidioanalyzer import AnalyzerEngine
analyzer = AnalyzerEngine()
text = """
Halo, nama saya John Smith dan email saya adalah john.smith@example.com.
Nomor telepon saya +62-812-3456-7890 dan nomor kartu kredit saya
4539-1488-0343-6467. Saya tinggal di Jakarta, Indonesia.
"""
results = analyzer.analyze(
text=text,
language="en",
entities=None # None = deteksi semua jenis entitas
)
for result in results:
print(f"Entitas: {result.entitytype}")
print(f" Teks: {text[result.start:result.end]}")
print(f" Skor: {result.score}")
print(f" Posisi: {result.start}-{result.end}")
print()
Output:
Entitas: PERSON
Teks: John Smith
Skor: 0.85
Posisi: 20-30
Entitas: EMAILADDRESS
Teks: john.smith@example.com
Skor: 1.0
Posisi: 51-73
Entitas: PHONENUMBER
Teks: +62-812-3456-7890
Skor: 0.75
Posisi: 95-113
Entitas: CREDITCARD
Teks: 4539-1488-0343-6467
Skor: 1.0
Posisi: 143-162
Entitas: LOCATION
Teks: Jakarta
Skor: 0.85
Posisi: 180-187
Mendeteksi Entitas Tertentu Saja
results = analyzer.analyze(
text=text,
language="en",
entities=["EMAILADDRESS", "PHONENUMBER", "CREDITCARD"]
)
for result in results:
print(f"{result.entitytype}: {text[result.start:result.end]} (skor: {result.score})")
Mengatur Threshold Skor
results = analyzer.analyze(
text=text,
language="en",
scorethreshold=0.7 # Hanya tampilkan hasil dengan skor >= 0.7
)
Anonimisasi Data
Anonimisasi Dasar (Replace)
from presidioanalyzer import AnalyzerEngine
from presidioanonymizer import AnonymizerEngine
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
text = "Hubungi John Smith di john@example.com atau +62-812-3456-7890"
results = analyzer.analyze(text=text, language="en")
anonymized = anonymizer.anonymize(text=text, analyzerresults=results)
print(anonymized.text)
Output: Hubungi di ADDRESS> atau NUMBER>
Anonimisasi dengan Strategi Berbeda
Presidio mendukung beberapa strategi anonimisasi:
from presidioanonymizer.entities import OperatorConfig
operators = {
"PERSON": OperatorConfig("replace", {"new
value": "NAMA DIRAHASIAKAN"}),
"EMAILADDRESS": OperatorConfig("mask", {"maskingchar": "", "charstomask": 8, "fromend": False}),
"PHONENUMBER": OperatorConfig("redact"),
"CREDITCARD": OperatorConfig("hash", {"hashtype": "sha256"}),
}
anonymized = anonymizer.anonymize(
text=text,
analyzerresults=results,
operators=operators
)
print(anonymized.text)
Jenis Operator Anonimisasi
| Operator | Deskripsi | Contoh |
|----------|-----------|--------|
| replace | Mengganti dengan teks tertentu | John Smith → |
| redact | Menghapus teks sepenuhnya | john@mail.com → ` |
| mask | Menutupi sebagian karakter | john@mail.com → **@mail.com |
| hash | Mengganti dengan hash | John → a8cfcd... |
| encrypt | Mengenkripsi teks | John → encryptedstring |
| custom` | Fungsi kustom | Sesuai kebutuhan |
Advanced Usage
Membuat Custom Recognizer
Anda bisa membuat recognizer kustom untuk mendeteksi format PII spesifik Indonesia, seperti NIK (Nomor Induk Kependudukan):
from presidioanalyzer import PatternRecognizer, Pattern
nik
pattern = Pattern(
name="nikpattern",
regex=r"\b\d{16}\b",
score=0.7
)
nikrecognizer = PatternRecognizer(
supportedentity="NIKINDONESIA",
name="Indonesian NIK Recognizer",
patterns=[nikpattern],
supportedlanguage="en",
context=["nik", "ktp", "identitas", "kependudukan"]
)
analyzer = AnalyzerEngine()
analyzer.registry.addrecognizer(nikrecognizer)
text = "NIK saya adalah 3201234567890001 dan ini data penting"
results = analyzer.analyze(text=text, language="en")
for result in results:
print(f"{result.entitytype}: {text[result.start:result.end]} (skor: {result.score})")
Recognizer untuk NPWP Indonesia
npwppattern = Pattern(
name="npwppattern",
regex=r"\b\d{2}\.\d{3}\.\d{3}\.\d{1}-\d{3}\.\d{3}\b",
score=0.85
)
npwprecognizer = PatternRecognizer(
supportedentity="NPWPINDONESIA",
name="Indonesian NPWP Recognizer",
patterns=[npwppattern],
supportedlanguage="en",
context=["npwp", "pajak", "tax"]
)
analyzer.registry.addrecognizer(npwprecognizer)
text = "NPWP perusahaan: 53.381.896.9-542.000"
results = analyzer.analyze(text=text, language="en")
for result in results:
print(f"{result.entitytype}: {text[result.start:result.end]}")
Output: NPWPINDONESIA: 53.381.896.9-542.000
Menggunakan Transformers untuk Akurasi Lebih Tinggi
from presidioanalyzer import AnalyzerEngine
from presidio
analyzer.nlpengine import TransformersNlpEngine
model
config = [{
"langcode": "en",
"modelname": {
"spacy": "encorewebsm",
"transformers": "dslim/bert-base-NER"
}
}]
nlpengine = TransformersNlpEngine(models=modelconfig)
analyzer = AnalyzerEngine(nlpengine=nlpengine)
text = "Dr. Sarah Connor bekerja di Cyberdyne Systems, email: sarah@cyberdyne.com"
results = analyzer.analyze(text=text, language="en")
for result in results:
print(f"{result.entitytype}: {text[result.start:result.end]} (skor: {result.score:.2f})")
Batch Processing untuk Dataset Besar
import pandas as pd
from presidioanalyzer import AnalyzerEngine, BatchAnalyzerEngine
from presidioanonymizer import AnonymizerEngine, BatchAnonymizerEngine
analyzer = AnalyzerEngine()
batchanalyzer = BatchAnalyzerEngine(analyzerengine=analyzer)
anonymizer = AnonymizerEngine()
batchanonymizer = BatchAnonymizerEngine(anonymizerengine=anonymizer)
data = pd.DataFrame({
"id": [1, 2, 3],
"feedback": [
"Nama saya Budi Santoso, email budi@gmail.com",
"Hubungi Ani di +62-811-2233-4455",
"Kartu kredit 4539-1488-0343-6467 bermasalah"
]
})
analyzerresults = batchanalyzer.analyzedict(
data.todict(orient="list"),
language="en",
keys=["feedback"]
)
anonymizedresults = batchanonymizer.anonymizedict(
analyzerresults
)
anonymizeddf = pd.DataFrame(anonymizedresults)
print(anonymizeddf)
Anonimisasi Data Terstruktur (JSON/Dict)
from presidioanalyzer import AnalyzerEngine
from presidioanonymizer import AnonymizerEngine
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
customerdata = {
"nama": "Ahmad Fadillah",
"email": "ahmad.fadillah@company.com",
"alamat": "Jl. Sudirman No. 52, Jakarta Selatan",
"telepon": "+62-851-8324-5226",
"catatan": "Customer VIP, hubungi via email ahmad.fadillah@company.com"
}
anonymizeddata = {}
for key, value in customerdata.items():
if isinstance(value, str):
results = analyzer.analyze(text=value, language="en")
if results:
anonymized = anonymizer.anonymize(text=value, analyzerresults=results)
anonymizeddata[key] = anonymized.text
else:
anonymizeddata[key] = value
import json
print(json.dumps(anonymizeddata, indent=2, ensureascii=False))
Deanonimisasi (Reversible Anonymization)
Dalam beberapa kasus, Anda perlu bisa mengembalikan data ke bentuk aslinya:
from presidioanonymizer import AnonymizerEngine, DeanonymizeEngine
from presidioanonymizer.entities import OperatorConfig
anonymizer = AnonymizerEngine()
deanonymizer = DeanonymizeEngine()
text = "Hubungi John Smith di john@example.com"
results = analyzer.analyze(text=text, language="en")
encryptionkey = "WmZq4t7w!z%C&F)J"
anonymized = anonymizer.anonymize(
text=text,
analyzerresults=results,
operators={
"DEFAULT": OperatorConfig("encrypt", {"key": encryptionkey})
}
)
print(f"Terenkripsi: {anonymized.text}")
deanonymized = deanonymizer.deanonymize(
text=anonymized.text,
entities=anonymized.items,
operators={
"DEFAULT": OperatorConfig("decrypt", {"key": encryptionkey})
}
)
print(f"Didekripsi: {deanonymized.text}")
Integrasi dengan Pipeline Data
Presidio + LangChain untuk RAG yang Aman
from presidioanalyzer import AnalyzerEngine
from presidioanonymizer import AnonymizerEngine
class PIISanitizer:
def init(self):
self.analyzer = AnalyzerEngine()
self.anonymizer = AnonymizerEngine()
def sanitize(self, text: str, language: str = "en") -> str:
results = self.analyzer.analyze(text=text, language=language)
if not results:
return text
anonymized = self.anonymizer.anonymize(text=text, analyzerresults=results)
return anonymized.text
def sanitizedocuments(self, documents: list[str]) -> list[str]:
return [self.sanitize(doc) for doc in documents]
sanitizer = PIISanitizer()
documents = [
"Laporan dibuat oleh Dr. Budi Hartono (budi@hospital.com) pada 15 Januari 2026",
"Pasien Ahmad (NIK: 3201234567890001) memiliki riwayat alergi",
"Pembayaran diterima dari kartu 4539-1488-0343-6467"
]
safedocuments = sanitizer.sanitizedocuments(documents)
for i, doc in enumerate(safedocuments):
print(f"Doc {i+1}: {doc}")
Presidio + FastAPI untuk API Anonimisasi
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from presidioanalyzer import AnalyzerEngine
from presidioanonymizer import AnonymizerEngine
app = FastAPI(title="PII Anonymization API")
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
class AnonymizeRequest(BaseModel):
text: str
language: str = "en"
entities: list[str] | None = None
scorethreshold: float = 0.5
class AnonymizeResponse(BaseModel):
originaltext: str
anonymizedtext: str
entitiesfound: list[dict]
@app.post("/anonymize", responsemodel=AnonymizeResponse)
def anonymizetext(request: AnonymizeRequest):
results = analyzer.analyze(
text=request.text,
language=request.language,
entities=request.entities,
scorethreshold=request.scorethreshold
)
entitiesfound = [
{
"type": r.entitytype,
"text": request.text[r.start:r.end],
"score": r.score,
"start": r.start,
"end": r.end
}
for r in results
]
anonymized = anonymizer.anonymize(text=request.text, analyzerresults=results)
return AnonymizeResponse(
originaltext=request.text,
anonymizedtext=anonymized.text,
entitiesfound=entitiesfound
)
Presidio + Pandas untuk Data Cleaning
import pandas as pd
from presidioanalyzer import AnalyzerEngine
from presidioanonymizer import AnonymizerEngine
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
def anonymizecolumn(series: pd.Series, language: str = "en") -> pd.Series:
def processtext(text):
if pd.isna(text) or not isinstance(text, str):
return text
results = analyzer.analyze(text=text, language=language)
if not results:
return text
return anonymizer.anonymize(text=text, analyzerresults=results).text
return series.apply(processtext)
df = pd.readcsv("customerfeedback.csv")
sensitivecolumns = ["nama", "komentar", "alamat", "catatan"]
for col in sensitivecolumns:
if col in df.columns:
df[col] = anonymizecolumn(df[col])
df.tocsv("customerfeedbackanonymized.csv", index=False)
print("Data berhasil dianonimisasi!")
Best Practices
1. Selalu Validasi Hasil Deteksi
Presidio mungkin menghasilkan false positive atau false negative. Selalu periksa hasil dengan threshold yang sesuai:
results = analyzer.analyze(
text=text,
language="en",
scorethreshold=0.7,
returndecisionprocess=True
)
for result in results:
analysis = result.analysisexplanation
print(f"Entitas: {result.entitytype}")
print(f" Recognizer: {analysis.recognizer}")
print(f" Pattern: {analysis.patternname}")
print(f" Skor asli: {analysis.score}")
print(f" Context enhancement: {analysis.scorecontextimprovement}")
2. Gunakan Allow List dan Deny List
from presidioanalyzer import AnalyzerEngine
analyzer = AnalyzerEngine()
text = "Hubungi tim support di support@company.com atau John Smith di john@personal.com"
allowlist = ["support@company.com"]
results = analyzer.analyze(
text=text,
language="en",
allowlist=allowlist
)
3. Logging dan Audit Trail
import logging
import json
from datetime import datetime
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("piiaudit")
def anonymizewithaudit(text: str, userid: str):
results = analyzer.analyze(text=text, language="en")
auditlog = {
"timestamp": datetime.now().isoformat(),
"userid": userid,
"entitiesdetected": len(results),
"entitytypes": list(set(r.entitytype for r in results)),
"textlength": len(text)
}
logger.info(f"PII Audit: {json.dumps(auditlog)}")
anonymized = anonymizer.anonymize(text=text, analyzerresults=results)
return anonymized.text
4. Performance Optimization untuk Data Besar
from concurrent.futures import ThreadPoolExecutor
from presidioanalyzer import AnalyzerEngine
from presidioanonymizer import AnonymizerEngine
def processbatch(texts: list[str], batchsize: int = 100):
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
results = []
for i in range(0, len(texts), batchsize):
batch = texts[i:i + batchsize]
for text in batch:
analysis = analyzer.analyze(text=text, language="en")
anonymized = anonymizer.anonymize(text=text, analyzerresults=analysis)
results.append(anonymized.text)
return results
texts = ["..."] 10000
anonymizedtexts = processbatch(texts, batchsize=500)
5. Testing dan Validasi
import pytest
from presidioanalyzer import AnalyzerEngine
analyzer = AnalyzerEngine()
def testemaildetection():
text = "Email saya test@example.com"
results = analyzer.analyze(text=text, language="en", entities=["EMAILADDRESS"])
assert len(results) == 1
assert results[0].entitytype == "EMAILADDRESS"
assert text[results[0].start:results[0].end] == "test@example.com"
def testphonedetection():
text = "Telepon: +62-812-3456-7890"
results = analyzer.analyze(text=text, language="en", entities=["PHONENUMBER"])
assert len(results) >= 1
def testnopii():
text = "Hari ini cuaca cerah dan sejuk"
results = analyzer.analyze(text=text, language="en")
assert len(results) == 0
def testcustomrecognizer():
nikrecognizer = PatternRecognizer(
supportedentity="NIKINDONESIA",
patterns=[Pattern("nik", r"\b\d{16}\b", 0.7)],
supportedlanguage="en",
context=["nik", "ktp"]
)
analyzer.registry.addrecognizer(nikrecognizer)
text = "NIK: 3201234567890001"
results = analyzer.analyze(text=text, language="en", entities=["NIKINDONESIA"])
assert len(results) == 1
Entitas yang Didukung
Presidio mendukung lebih dari 20 jenis entitas PII secara default:
| Entitas | Deskripsi |
|---------|-----------|
| PERSON | Nama orang |
| EMAILADDRESS | Alamat email |
| PHONENUMBER | Nomor telepon |
| CREDITCARD | Nomor kartu kredit |
| IBANCODE | Kode IBAN bank |
| IPADDRESS | Alamat IP |
| LOCATION | Lokasi/alamat |
| DATETIME | Tanggal dan waktu |
| NRP | Nomor registrasi |
| MEDICALLICENSE | Lisensi medis |
| URL | Alamat URL |
| USSSN | Social Security Number (US) |
| USPASSPORT | Nomor paspor (US) |
| USDRIVERLICENSE | SIM (US) |
| CRYPTO | Alamat cryptocurrency |
| UKNHS | NHS number (UK) |
| SGNRICFIN | NRIC/FIN (Singapore) |
| AU_ABN | ABN (Australia) |
Dan Anda bisa menambahkan recognizer kustom untuk format lokal seperti NIK, NPWP, dan SIM Indonesia.
Kesimpulan
Microsoft Presidio adalah tools yang sangat berguna untuk setiap developer dan data engineer yang bekerja dengan data sensitif. Dengan kombinasi NER berbasis AI dan pattern matching, Presidio memberikan fleksibilitas tinggi untuk mendeteksi dan menganonimisasi berbagai jenis PII.
Poin-poin penting yang perlu diingat:
Dengan meningkatnya regulasi privasi data di Indonesia dan global, kemampuan mengelola PII secara otomatis menjadi skill yang sangat berharga. Presidio menyediakan fondasi yang kuat untuk membangun sistem perlindungan data yang reliable dan scalable.