Presidio Tutorial: Detect and Anonymize PII Data with Python
In the age of AI and big data, protecting personally identifiable information (PII) is no longer optional but mandatory. Regulations like GDPR in Europe, CCPA in California, and Indonesia's UU PDP require organizations to handle personal data responsibly. Microsoft Presidio is a powerful open-source SDK for detecting, analyzing, and anonymizing PII in text and structured data.
This tutorial walks you through everything from installation to advanced usage, with practical code examples you can run immediately in your projects.
What Is Microsoft Presidio?
Presidio is an open-source SDK from Microsoft that provides modules for detecting and anonymizing sensitive data. It supports various PII types such as names, emails, phone numbers, national IDs, credit cards, and more. Presidio consists of two main components:
- Presidio Analyzer: Detects and classifies PII entities in text
- Presidio Anonymizer: Anonymizes detected PII entities using various strategies
Presidio uses a combination of Named Entity Recognition (NER) powered by SpaCy/Transformers, regular expressions, and rule-based logic to detect PII with high accuracy.
Installation
Basic Installation
pip install presidio-analyzer presidio-anonymizer
Installing SpaCy Models
Presidio requires NLP models for entity detection. For English:
python -m spacy download encoreweblg
For multilingual support:
python -m spacy download xxentwikism
Full Installation (with Transformers)
For higher accuracy using Transformer models:
pip install presidio-analyzer presidio-anonymizer torch transformers
python -m spacy download encoreweblg
Verify Installation
from presidioanalyzer import AnalyzerEngine
from presidioanonymizer import AnonymizerEngine
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
print("Presidio installed successfully!")
Basic Usage: Detecting PII
Detecting PII in Text
from presidioanalyzer import AnalyzerEngine
analyzer = AnalyzerEngine()
text = """
Hello, my name is John Smith and my email is john.smith@example.com.
My phone number is +1-555-123-4567 and my credit card number is
4539-1488-0343-6467. I live in New York City.
"""
results = analyzer.analyze(
text=text,
language="en",
entities=None # None = detect all entity types
)
for result in results:
print(f"Entity: {result.entitytype}")
print(f" Text: {text[result.start:result.end]}")
print(f" Score: {result.score}")
print(f" Position: {result.start}-{result.end}")
print()
Output:
Entity: PERSON
Text: John Smith
Score: 0.85
Position: 20-30
Entity: EMAILADDRESS
Text: john.smith@example.com
Score: 1.0
Position: 51-73
Entity: PHONENUMBER
Text: +1-555-123-4567
Score: 0.75
Position: 95-111
Entity: CREDITCARD
Text: 4539-1488-0343-6467
Score: 1.0
Position: 141-160
Entity: LOCATION
Text: New York City
Score: 0.85
Position: 176-189
Detecting Specific Entities Only
results = analyzer.analyze(
text=text,
language="en",
entities=["EMAILADDRESS", "PHONENUMBER", "CREDITCARD"]
)
for result in results:
print(f"{result.entitytype}: {text[result.start:result.end]} (score: {result.score})")
Setting Score Threshold
results = analyzer.analyze(
text=text,
language="en",
scorethreshold=0.7 # Only show results with score >= 0.7
)
Anonymizing Data
Basic Anonymization (Replace)
from presidioanalyzer import AnalyzerEngine
from presidioanonymizer import AnonymizerEngine
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
text = "Contact John Smith at john@example.com or +1-555-123-4567"
results = analyzer.analyze(text=text, language="en")
anonymized = anonymizer.anonymize(text=text, analyzerresults=results)
print(anonymized.text)
Output: Contact at ADDRESS> or NUMBER>
Anonymization with Different Strategies
Presidio supports several anonymization strategies:
from presidioanonymizer.entities import OperatorConfig
operators = {
"PERSON": OperatorConfig("replace", {"new
value": "REDACTED NAME"}),
"EMAILADDRESS": OperatorConfig("mask", {"maskingchar": "", "charstomask": 8, "fromend": False}),
"PHONENUMBER": OperatorConfig("redact"),
"CREDITCARD": OperatorConfig("hash", {"hashtype": "sha256"}),
}
anonymized = anonymizer.anonymize(
text=text,
analyzerresults=results,
operators=operators
)
print(anonymized.text)
Anonymization Operator Types
| Operator | Description | Example |
|----------|-------------|---------|
| replace | Replace with specific text | John Smith → |
| redact | Remove text entirely | john@mail.com → ` |
| mask | Mask partial characters | john@mail.com → **@mail.com |
| hash | Replace with hash value | John → a8cfcd... |
| encrypt | Encrypt the text | John → encryptedstring |
| custom` | Custom function | As needed |
Advanced Usage
Creating Custom Recognizers
You can build custom recognizers to detect country-specific PII formats. Here is an example for Indonesian NIK (National ID Number):
from presidioanalyzer import PatternRecognizer, Pattern
nik
pattern = Pattern(
name="nikpattern",
regex=r"\b\d{16}\b",
score=0.7
)
nikrecognizer = PatternRecognizer(
supportedentity="NIKINDONESIA",
name="Indonesian NIK Recognizer",
patterns=[nikpattern],
supportedlanguage="en",
context=["nik", "ktp", "identity", "national id"]
)
analyzer = AnalyzerEngine()
analyzer.registry.addrecognizer(nikrecognizer)
text = "My NIK is 3201234567890001 and this is important data"
results = analyzer.analyze(text=text, language="en")
for result in results:
print(f"{result.entitytype}: {text[result.start:result.end]} (score: {result.score})")
Custom Recognizer for Indonesian NPWP (Tax ID)
npwppattern = Pattern(
name="npwppattern",
regex=r"\b\d{2}\.\d{3}\.\d{3}\.\d{1}-\d{3}\.\d{3}\b",
score=0.85
)
npwprecognizer = PatternRecognizer(
supportedentity="NPWPINDONESIA",
name="Indonesian NPWP Recognizer",
patterns=[npwppattern],
supportedlanguage="en",
context=["npwp", "tax", "pajak"]
)
analyzer.registry.addrecognizer(npwprecognizer)
text = "Company NPWP: 53.381.896.9-542.000"
results = analyzer.analyze(text=text, language="en")
for result in results:
print(f"{result.entitytype}: {text[result.start:result.end]}")
Output: NPWPINDONESIA: 53.381.896.9-542.000
Using Transformers for Higher Accuracy
from presidioanalyzer import AnalyzerEngine
from presidio
analyzer.nlpengine import TransformersNlpEngine
model
config = [{
"langcode": "en",
"modelname": {
"spacy": "encorewebsm",
"transformers": "dslim/bert-base-NER"
}
}]
nlpengine = TransformersNlpEngine(models=modelconfig)
analyzer = AnalyzerEngine(nlpengine=nlpengine)
text = "Dr. Sarah Connor works at Cyberdyne Systems, email: sarah@cyberdyne.com"
results = analyzer.analyze(text=text, language="en")
for result in results:
print(f"{result.entitytype}: {text[result.start:result.end]} (score: {result.score:.2f})")
Batch Processing for Large Datasets
import pandas as pd
from presidioanalyzer import AnalyzerEngine, BatchAnalyzerEngine
from presidioanonymizer import AnonymizerEngine, BatchAnonymizerEngine
analyzer = AnalyzerEngine()
batchanalyzer = BatchAnalyzerEngine(analyzerengine=analyzer)
anonymizer = AnonymizerEngine()
batchanonymizer = BatchAnonymizerEngine(anonymizerengine=anonymizer)
data = pd.DataFrame({
"id": [1, 2, 3],
"feedback": [
"My name is John Doe, email john@gmail.com",
"Contact Sarah at +1-555-987-6543",
"Credit card 4539-1488-0343-6467 has issues"
]
})
analyzerresults = batchanalyzer.analyzedict(
data.todict(orient="list"),
language="en",
keys=["feedback"]
)
anonymizedresults = batchanonymizer.anonymizedict(
analyzerresults
)
anonymizeddf = pd.DataFrame(anonymizedresults)
print(anonymizeddf)
Anonymizing Structured Data (JSON/Dict)
from presidioanalyzer import AnalyzerEngine
from presidioanonymizer import AnonymizerEngine
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
customerdata = {
"name": "Ahmad Fadillah",
"email": "ahmad.fadillah@company.com",
"address": "52 Sudirman Street, South Jakarta",
"phone": "+62-851-8324-5226",
"notes": "VIP customer, contact via email ahmad.fadillah@company.com"
}
anonymizeddata = {}
for key, value in customerdata.items():
if isinstance(value, str):
results = analyzer.analyze(text=value, language="en")
if results:
anonymized = anonymizer.anonymize(text=value, analyzerresults=results)
anonymizeddata[key] = anonymized.text
else:
anonymizeddata[key] = value
import json
print(json.dumps(anonymizeddata, indent=2))
Reversible Anonymization (Deanonymization)
In some cases, you need to be able to restore data to its original form:
from presidioanonymizer import AnonymizerEngine, DeanonymizeEngine
from presidio
anonymizer.entities import OperatorConfig
anonymizer = AnonymizerEngine()
deanonymizer = DeanonymizeEngine()
text = "Contact John Smith at john@example.com"
results = analyzer.analyze(text=text, language="en")
encryptionkey = "WmZq4t7w!z%C&F)J"
anonymized = anonymizer.anonymize(
text=text,
analyzerresults=results,
operators={
"DEFAULT": OperatorConfig("encrypt", {"key": encryptionkey})
}
)
print(f"Encrypted: {anonymized.text}")
deanonymized = deanonymizer.deanonymize(
text=anonymized.text,
entities=anonymized.items,
operators={
"DEFAULT": OperatorConfig("decrypt", {"key": encryptionkey})
}
)
print(f"Decrypted: {deanonymized.text}")
Integration with Data Pipelines
Presidio + LangChain for Safe RAG
from presidioanalyzer import AnalyzerEngine
from presidio
anonymizer import AnonymizerEngine
class PIISanitizer:
def init(self):
self.analyzer = AnalyzerEngine()
self.anonymizer = AnonymizerEngine()
def sanitize(self, text: str, language: str = "en") -> str:
results = self.analyzer.analyze(text=text, language=language)
if not results:
return text
anonymized = self.anonymizer.anonymize(text=text, analyzerresults=results)
return anonymized.text
def sanitizedocuments(self, documents: list[str]) -> list[str]:
return [self.sanitize(doc) for doc in documents]
sanitizer = PIISanitizer()
documents = [
"Report by Dr. John Hartono (john@hospital.com) on January 15, 2026",
"Patient Ahmad (ID: 3201234567890001) has allergy history",
"Payment received from card 4539-1488-0343-6467"
]
safedocuments = sanitizer.sanitizedocuments(documents)
for i, doc in enumerate(safedocuments):
print(f"Doc {i+1}: {doc}")
Presidio + FastAPI for an Anonymization API
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from presidioanalyzer import AnalyzerEngine
from presidioanonymizer import AnonymizerEngine
app = FastAPI(title="PII Anonymization API")
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
class AnonymizeRequest(BaseModel):
text: str
language: str = "en"
entities: list[str] | None = None
scorethreshold: float = 0.5
class AnonymizeResponse(BaseModel):
originaltext: str
anonymizedtext: str
entitiesfound: list[dict]
@app.post("/anonymize", responsemodel=AnonymizeResponse)
def anonymizetext(request: AnonymizeRequest):
results = analyzer.analyze(
text=request.text,
language=request.language,
entities=request.entities,
scorethreshold=request.scorethreshold
)
entitiesfound = [
{
"type": r.entitytype,
"text": request.text[r.start:r.end],
"score": r.score,
"start": r.start,
"end": r.end
}
for r in results
]
anonymized = anonymizer.anonymize(text=request.text, analyzerresults=results)
return AnonymizeResponse(
originaltext=request.text,
anonymizedtext=anonymized.text,
entitiesfound=entitiesfound
)
Presidio + Pandas for Data Cleaning
import pandas as pd
from presidioanalyzer import AnalyzerEngine
from presidioanonymizer import AnonymizerEngine
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
def anonymizecolumn(series: pd.Series, language: str = "en") -> pd.Series:
def processtext(text):
if pd.isna(text) or not isinstance(text, str):
return text
results = analyzer.analyze(text=text, language=language)
if not results:
return text
return anonymizer.anonymize(text=text, analyzerresults=results).text
return series.apply(processtext)
df = pd.readcsv("customerfeedback.csv")
sensitivecolumns = ["name", "comment", "address", "notes"]
for col in sensitivecolumns:
if col in df.columns:
df[col] = anonymizecolumn(df[col])
df.tocsv("customerfeedbackanonymized.csv", index=False)
print("Data anonymized successfully!")
Best Practices
1. Always Validate Detection Results
Presidio may produce false positives or false negatives. Always verify results with appropriate thresholds:
results = analyzer.analyze(
text=text,
language="en",
scorethreshold=0.7,
returndecisionprocess=True
)
for result in results:
analysis = result.analysisexplanation
print(f"Entity: {result.entitytype}")
print(f" Recognizer: {analysis.recognizer}")
print(f" Pattern: {analysis.patternname}")
print(f" Original score: {analysis.score}")
print(f" Context enhancement: {analysis.scorecontextimprovement}")
2. Use Allow Lists and Deny Lists
from presidioanalyzer import AnalyzerEngine
analyzer = AnalyzerEngine()
text = "Contact the support team at support@company.com or John Smith at john@personal.com"
allow
list = ["support@company.com"]
results = analyzer.analyze(
text=text,
language="en",
allowlist=allowlist
)
3. Logging and Audit Trails
import logging
import json
from datetime import datetime
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("piiaudit")
def anonymizewithaudit(text: str, userid: str):
results = analyzer.analyze(text=text, language="en")
auditlog = {
"timestamp": datetime.now().isoformat(),
"userid": userid,
"entitiesdetected": len(results),
"entitytypes": list(set(r.entitytype for r in results)),
"textlength": len(text)
}
logger.info(f"PII Audit: {json.dumps(auditlog)}")
anonymized = anonymizer.anonymize(text=text, analyzerresults=results)
return anonymized.text
4. Performance Optimization for Large Data
from concurrent.futures import ThreadPoolExecutor
from presidioanalyzer import AnalyzerEngine
from presidioanonymizer import AnonymizerEngine
def processbatch(texts: list[str], batchsize: int = 100):
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
results = []
for i in range(0, len(texts), batchsize):
batch = texts[i:i + batchsize]
for text in batch:
analysis = analyzer.analyze(text=text, language="en")
anonymized = anonymizer.anonymize(text=text, analyzerresults=analysis)
results.append(anonymized.text)
return results
texts = ["..."] 10000
anonymizedtexts = processbatch(texts, batchsize=500)
5. Testing and Validation
import pytest
from presidioanalyzer import AnalyzerEngine
analyzer = AnalyzerEngine()
def testemaildetection():
text = "My email is test@example.com"
results = analyzer.analyze(text=text, language="en", entities=["EMAILADDRESS"])
assert len(results) == 1
assert results[0].entitytype == "EMAILADDRESS"
assert text[results[0].start:results[0].end] == "test@example.com"
def testphonedetection():
text = "Phone: +1-555-123-4567"
results = analyzer.analyze(text=text, language="en", entities=["PHONENUMBER"])
assert len(results) >= 1
def testnopii():
text = "The weather is sunny and cool today"
results = analyzer.analyze(text=text, language="en")
assert len(results) == 0
def testcustomrecognizer():
nikrecognizer = PatternRecognizer(
supportedentity="NIKINDONESIA",
patterns=[Pattern("nik", r"\b\d{16}\b", 0.7)],
supportedlanguage="en",
context=["nik", "ktp"]
)
analyzer.registry.addrecognizer(nikrecognizer)
text = "NIK: 3201234567890001"
results = analyzer.analyze(text=text, language="en", entities=["NIKINDONESIA"])
assert len(results) == 1
Supported Entities
Presidio supports over 20 PII entity types out of the box:
| Entity | Description |
|--------|-------------|
| PERSON | Person names |
| EMAILADDRESS | Email addresses |
| PHONENUMBER | Phone numbers |
| CREDITCARD | Credit card numbers |
| IBANCODE | Bank IBAN codes |
| IPADDRESS | IP addresses |
| LOCATION | Locations/addresses |
| DATETIME | Dates and times |
| NRP | Registration numbers |
| MEDICALLICENSE | Medical licenses |
| URL | Web URLs |
| USSSN | Social Security Numbers (US) |
| USPASSPORT | Passport numbers (US) |
| USDRIVERLICENSE | Driver licenses (US) |
| CRYPTO | Cryptocurrency addresses |
| UKNHS | NHS numbers (UK) |
| SGNRICFIN | NRIC/FIN (Singapore) |
| AUABN | ABN (Australia) |
You can extend this by adding custom recognizers for local formats like Indonesian NIK, NPWP, and SIM.
Conclusion
Microsoft Presidio is an essential tool for every developer and data engineer working with sensitive data. With its combination of AI-powered NER and pattern matching, Presidio provides high flexibility for detecting and anonymizing various PII types.
Key takeaways:
As data privacy regulations continue to tighten worldwide, the ability to automatically manage PII is becoming an invaluable skill. Presidio provides a robust foundation for building reliable and scalable data protection systems.