Presidio Tutorial: Detect and Anonymize PII Data with Python

# Tutorial Presidio: Deteksi dan Anonimisasi Data Pribadi (PII) dengan Python Dalam era AI dan big data, perlindungan data pribadi bukan lagi pilihan melainkan keharusan. Regulasi seperti UU PDP (Und...

By Ruby Abdullah · · tutorial
PresidioPIIData PrivacyPythonNLP

Presidio Tutorial: Detect and Anonymize PII Data with Python

In the age of AI and big data, protecting personally identifiable information (PII) is no longer optional but mandatory. Regulations like GDPR in Europe, CCPA in California, and Indonesia's UU PDP require organizations to handle personal data responsibly. Microsoft Presidio is a powerful open-source SDK for detecting, analyzing, and anonymizing PII in text and structured data.

This tutorial walks you through everything from installation to advanced usage, with practical code examples you can run immediately in your projects.

What Is Microsoft Presidio?

Presidio is an open-source SDK from Microsoft that provides modules for detecting and anonymizing sensitive data. It supports various PII types such as names, emails, phone numbers, national IDs, credit cards, and more. Presidio consists of two main components:

  • Presidio Analyzer: Detects and classifies PII entities in text
  • Presidio Anonymizer: Anonymizes detected PII entities using various strategies

Presidio uses a combination of Named Entity Recognition (NER) powered by SpaCy/Transformers, regular expressions, and rule-based logic to detect PII with high accuracy.

Installation

Basic Installation

pip install presidio-analyzer presidio-anonymizer

Installing SpaCy Models

Presidio requires NLP models for entity detection. For English:

python -m spacy download encoreweblg

For multilingual support:

python -m spacy download xxentwikism

Full Installation (with Transformers)

For higher accuracy using Transformer models:

pip install presidio-analyzer presidio-anonymizer torch transformers

python -m spacy download encoreweblg

Verify Installation

from presidioanalyzer import AnalyzerEngine

from presidioanonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()

anonymizer = AnonymizerEngine()

print("Presidio installed successfully!")

Basic Usage: Detecting PII

Detecting PII in Text

from presidioanalyzer import AnalyzerEngine

analyzer = AnalyzerEngine()

text = """

Hello, my name is John Smith and my email is john.smith@example.com.

My phone number is +1-555-123-4567 and my credit card number is

4539-1488-0343-6467. I live in New York City.

"""

results = analyzer.analyze(

text=text,

language="en",

entities=None # None = detect all entity types

)

for result in results:

print(f"Entity: {result.entitytype}")

print(f" Text: {text[result.start:result.end]}")

print(f" Score: {result.score}")

print(f" Position: {result.start}-{result.end}")

print()

Output:

Entity: PERSON

Text: John Smith

Score: 0.85

Position: 20-30

Entity: EMAILADDRESS

Text: john.smith@example.com

Score: 1.0

Position: 51-73

Entity: PHONENUMBER

Text: +1-555-123-4567

Score: 0.75

Position: 95-111

Entity: CREDITCARD

Text: 4539-1488-0343-6467

Score: 1.0

Position: 141-160

Entity: LOCATION

Text: New York City

Score: 0.85

Position: 176-189

Detecting Specific Entities Only

results = analyzer.analyze(

text=text,

language="en",

entities=["EMAILADDRESS", "PHONENUMBER", "CREDITCARD"]

)

for result in results:

print(f"{result.entitytype}: {text[result.start:result.end]} (score: {result.score})")

Setting Score Threshold

results = analyzer.analyze(

text=text,

language="en",

scorethreshold=0.7 # Only show results with score >= 0.7

)

Anonymizing Data

Basic Anonymization (Replace)

from presidioanalyzer import AnalyzerEngine

from presidioanonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()

anonymizer = AnonymizerEngine()

text = "Contact John Smith at john@example.com or +1-555-123-4567"

results = analyzer.analyze(text=text, language="en")

anonymized = anonymizer.anonymize(text=text, analyzerresults=results)

print(anonymized.text)

Output: Contact at ADDRESS> or NUMBER>

Anonymization with Different Strategies

Presidio supports several anonymization strategies:

from presidioanonymizer.entities import OperatorConfig

operators = {

"PERSON": OperatorConfig("replace", {"newvalue": "REDACTED NAME"}),

"EMAILADDRESS": OperatorConfig("mask", {"maskingchar": "", "charstomask": 8, "fromend": False}),

"PHONENUMBER": OperatorConfig("redact"),

"CREDITCARD": OperatorConfig("hash", {"hashtype": "sha256"}),

}

anonymized = anonymizer.anonymize(

text=text,

analyzerresults=results,

operators=operators

)

print(anonymized.text)

Anonymization Operator Types

| Operator | Description | Example |

|----------|-------------|---------|

| replace | Replace with specific text | John Smith → |

| redact | Remove text entirely | john@mail.com → ` |

| mask | Mask partial characters | john@mail.com → **@mail.com |

| hash | Replace with hash value | John → a8cfcd... |

| encrypt | Encrypt the text | John → encryptedstring |

| custom` | Custom function | As needed |

Advanced Usage

Creating Custom Recognizers

You can build custom recognizers to detect country-specific PII formats. Here is an example for Indonesian NIK (National ID Number):

from presidioanalyzer import PatternRecognizer, Pattern

nikpattern = Pattern(

name="nikpattern",

regex=r"\b\d{16}\b",

score=0.7

)

nikrecognizer = PatternRecognizer(

supportedentity="NIKINDONESIA",

name="Indonesian NIK Recognizer",

patterns=[nikpattern],

supportedlanguage="en",

context=["nik", "ktp", "identity", "national id"]

)

analyzer = AnalyzerEngine()

analyzer.registry.addrecognizer(nikrecognizer)

text = "My NIK is 3201234567890001 and this is important data"

results = analyzer.analyze(text=text, language="en")

for result in results:

print(f"{result.entitytype}: {text[result.start:result.end]} (score: {result.score})")

Custom Recognizer for Indonesian NPWP (Tax ID)

npwppattern = Pattern(

name="npwppattern",

regex=r"\b\d{2}\.\d{3}\.\d{3}\.\d{1}-\d{3}\.\d{3}\b",

score=0.85

)

npwprecognizer = PatternRecognizer(

supportedentity="NPWPINDONESIA",

name="Indonesian NPWP Recognizer",

patterns=[npwppattern],

supportedlanguage="en",

context=["npwp", "tax", "pajak"]

)

analyzer.registry.addrecognizer(npwprecognizer)

text = "Company NPWP: 53.381.896.9-542.000"

results = analyzer.analyze(text=text, language="en")

for result in results:

print(f"{result.entitytype}: {text[result.start:result.end]}")

Output: NPWPINDONESIA: 53.381.896.9-542.000

Using Transformers for Higher Accuracy

from presidioanalyzer import AnalyzerEngine

from presidioanalyzer.nlpengine import TransformersNlpEngine

modelconfig = [{

"langcode": "en",

"modelname": {

"spacy": "encorewebsm",

"transformers": "dslim/bert-base-NER"

}

}]

nlpengine = TransformersNlpEngine(models=modelconfig)

analyzer = AnalyzerEngine(nlpengine=nlpengine)

text = "Dr. Sarah Connor works at Cyberdyne Systems, email: sarah@cyberdyne.com"

results = analyzer.analyze(text=text, language="en")

for result in results:

print(f"{result.entitytype}: {text[result.start:result.end]} (score: {result.score:.2f})")

Batch Processing for Large Datasets

import pandas as pd

from presidioanalyzer import AnalyzerEngine, BatchAnalyzerEngine

from presidioanonymizer import AnonymizerEngine, BatchAnonymizerEngine

analyzer = AnalyzerEngine()

batchanalyzer = BatchAnalyzerEngine(analyzerengine=analyzer)

anonymizer = AnonymizerEngine()

batchanonymizer = BatchAnonymizerEngine(anonymizerengine=anonymizer)

data = pd.DataFrame({

"id": [1, 2, 3],

"feedback": [

"My name is John Doe, email john@gmail.com",

"Contact Sarah at +1-555-987-6543",

"Credit card 4539-1488-0343-6467 has issues"

]

})

analyzerresults = batchanalyzer.analyzedict(

data.todict(orient="list"),

language="en",

keys=["feedback"]

)

anonymizedresults = batchanonymizer.anonymizedict(

analyzerresults

)

anonymizeddf = pd.DataFrame(anonymizedresults)

print(anonymizeddf)

Anonymizing Structured Data (JSON/Dict)

from presidioanalyzer import AnalyzerEngine

from presidioanonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()

anonymizer = AnonymizerEngine()

customerdata = {

"name": "Ahmad Fadillah",

"email": "ahmad.fadillah@company.com",

"address": "52 Sudirman Street, South Jakarta",

"phone": "+62-851-8324-5226",

"notes": "VIP customer, contact via email ahmad.fadillah@company.com"

}

anonymizeddata = {}

for key, value in customerdata.items():

if isinstance(value, str):

results = analyzer.analyze(text=value, language="en")

if results:

anonymized = anonymizer.anonymize(text=value, analyzerresults=results)

anonymizeddata[key] = anonymized.text

else:

anonymizeddata[key] = value

import json

print(json.dumps(anonymizeddata, indent=2))

Reversible Anonymization (Deanonymization)

In some cases, you need to be able to restore data to its original form:

from presidioanonymizer import AnonymizerEngine, DeanonymizeEngine

from presidioanonymizer.entities import OperatorConfig

anonymizer = AnonymizerEngine()

deanonymizer = DeanonymizeEngine()

text = "Contact John Smith at john@example.com"

results = analyzer.analyze(text=text, language="en")

encryptionkey = "WmZq4t7w!z%C&F)J"

anonymized = anonymizer.anonymize(

text=text,

analyzerresults=results,

operators={

"DEFAULT": OperatorConfig("encrypt", {"key": encryptionkey})

}

)

print(f"Encrypted: {anonymized.text}")

deanonymized = deanonymizer.deanonymize(

text=anonymized.text,

entities=anonymized.items,

operators={

"DEFAULT": OperatorConfig("decrypt", {"key": encryptionkey})

}

)

print(f"Decrypted: {deanonymized.text}")

Integration with Data Pipelines

Presidio + LangChain for Safe RAG

from presidioanalyzer import AnalyzerEngine

from presidioanonymizer import AnonymizerEngine

class PIISanitizer:

def init(self):

self.analyzer = AnalyzerEngine()

self.anonymizer = AnonymizerEngine()

def sanitize(self, text: str, language: str = "en") -> str:

results = self.analyzer.analyze(text=text, language=language)

if not results:

return text

anonymized = self.anonymizer.anonymize(text=text, analyzerresults=results)

return anonymized.text

def sanitizedocuments(self, documents: list[str]) -> list[str]:

return [self.sanitize(doc) for doc in documents]

sanitizer = PIISanitizer()

documents = [

"Report by Dr. John Hartono (john@hospital.com) on January 15, 2026",

"Patient Ahmad (ID: 3201234567890001) has allergy history",

"Payment received from card 4539-1488-0343-6467"

]

safedocuments = sanitizer.sanitizedocuments(documents)

for i, doc in enumerate(safedocuments):

print(f"Doc {i+1}: {doc}")

Presidio + FastAPI for an Anonymization API

from fastapi import FastAPI, HTTPException

from pydantic import BaseModel

from presidioanalyzer import AnalyzerEngine

from presidioanonymizer import AnonymizerEngine

app = FastAPI(title="PII Anonymization API")

analyzer = AnalyzerEngine()

anonymizer = AnonymizerEngine()

class AnonymizeRequest(BaseModel):

text: str

language: str = "en"

entities: list[str] | None = None

scorethreshold: float = 0.5

class AnonymizeResponse(BaseModel):

originaltext: str

anonymizedtext: str

entitiesfound: list[dict]

@app.post("/anonymize", responsemodel=AnonymizeResponse)

def anonymizetext(request: AnonymizeRequest):

results = analyzer.analyze(

text=request.text,

language=request.language,

entities=request.entities,

scorethreshold=request.scorethreshold

)

entitiesfound = [

{

"type": r.entitytype,

"text": request.text[r.start:r.end],

"score": r.score,

"start": r.start,

"end": r.end

}

for r in results

]

anonymized = anonymizer.anonymize(text=request.text, analyzerresults=results)

return AnonymizeResponse(

originaltext=request.text,

anonymizedtext=anonymized.text,

entitiesfound=entitiesfound

)

Presidio + Pandas for Data Cleaning

import pandas as pd

from presidioanalyzer import AnalyzerEngine

from presidioanonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()

anonymizer = AnonymizerEngine()

def anonymizecolumn(series: pd.Series, language: str = "en") -> pd.Series:

def processtext(text):

if pd.isna(text) or not isinstance(text, str):

return text

results = analyzer.analyze(text=text, language=language)

if not results:

return text

return anonymizer.anonymize(text=text, analyzerresults=results).text

return series.apply(processtext)

df = pd.readcsv("customerfeedback.csv")

sensitivecolumns = ["name", "comment", "address", "notes"]

for col in sensitivecolumns:

if col in df.columns:

df[col] = anonymizecolumn(df[col])

df.tocsv("customerfeedbackanonymized.csv", index=False)

print("Data anonymized successfully!")

Best Practices

1. Always Validate Detection Results

Presidio may produce false positives or false negatives. Always verify results with appropriate thresholds:

results = analyzer.analyze(

text=text,

language="en",

scorethreshold=0.7,

returndecisionprocess=True

)

for result in results:

analysis = result.analysisexplanation

print(f"Entity: {result.entitytype}")

print(f" Recognizer: {analysis.recognizer}")

print(f" Pattern: {analysis.patternname}")

print(f" Original score: {analysis.score}")

print(f" Context enhancement: {analysis.scorecontextimprovement}")

2. Use Allow Lists and Deny Lists

from presidioanalyzer import AnalyzerEngine

analyzer = AnalyzerEngine()

text = "Contact the support team at support@company.com or John Smith at john@personal.com"

allowlist = ["support@company.com"]

results = analyzer.analyze(

text=text,

language="en",

allowlist=allowlist

)

3. Logging and Audit Trails

import logging

import json

from datetime import datetime

logging.basicConfig(level=logging.INFO)

logger = logging.getLogger("piiaudit")

def anonymizewithaudit(text: str, userid: str):

results = analyzer.analyze(text=text, language="en")

auditlog = {

"timestamp": datetime.now().isoformat(),

"userid": userid,

"entitiesdetected": len(results),

"entitytypes": list(set(r.entitytype for r in results)),

"textlength": len(text)

}

logger.info(f"PII Audit: {json.dumps(auditlog)}")

anonymized = anonymizer.anonymize(text=text, analyzerresults=results)

return anonymized.text

4. Performance Optimization for Large Data

from concurrent.futures import ThreadPoolExecutor

from presidioanalyzer import AnalyzerEngine

from presidioanonymizer import AnonymizerEngine

def processbatch(texts: list[str], batchsize: int = 100):

analyzer = AnalyzerEngine()

anonymizer = AnonymizerEngine()

results = []

for i in range(0, len(texts), batchsize):

batch = texts[i:i + batchsize]

for text in batch:

analysis = analyzer.analyze(text=text, language="en")

anonymized = anonymizer.anonymize(text=text, analyzerresults=analysis)

results.append(anonymized.text)

return results

texts = ["..."] 10000

anonymizedtexts = processbatch(texts, batchsize=500)

5. Testing and Validation

import pytest

from presidioanalyzer import AnalyzerEngine

analyzer = AnalyzerEngine()

def testemaildetection():

text = "My email is test@example.com"

results = analyzer.analyze(text=text, language="en", entities=["EMAILADDRESS"])

assert len(results) == 1

assert results[0].entitytype == "EMAILADDRESS"

assert text[results[0].start:results[0].end] == "test@example.com"

def testphonedetection():

text = "Phone: +1-555-123-4567"

results = analyzer.analyze(text=text, language="en", entities=["PHONENUMBER"])

assert len(results) >= 1

def testnopii():

text = "The weather is sunny and cool today"

results = analyzer.analyze(text=text, language="en")

assert len(results) == 0

def testcustomrecognizer():

nikrecognizer = PatternRecognizer(

supportedentity="NIKINDONESIA",

patterns=[Pattern("nik", r"\b\d{16}\b", 0.7)],

supportedlanguage="en",

context=["nik", "ktp"]

)

analyzer.registry.addrecognizer(nikrecognizer)

text = "NIK: 3201234567890001"

results = analyzer.analyze(text=text, language="en", entities=["NIKINDONESIA"])

assert len(results) == 1

Supported Entities

Presidio supports over 20 PII entity types out of the box:

| Entity | Description |

|--------|-------------|

| PERSON | Person names |

| EMAILADDRESS | Email addresses |

| PHONENUMBER | Phone numbers |

| CREDITCARD | Credit card numbers |

| IBANCODE | Bank IBAN codes |

| IPADDRESS | IP addresses |

| LOCATION | Locations/addresses |

| DATETIME | Dates and times |

| NRP | Registration numbers |

| MEDICALLICENSE | Medical licenses |

| URL | Web URLs |

| USSSN | Social Security Numbers (US) |

| USPASSPORT | Passport numbers (US) |

| USDRIVERLICENSE | Driver licenses (US) |

| CRYPTO | Cryptocurrency addresses |

| UKNHS | NHS numbers (UK) |

| SGNRICFIN | NRIC/FIN (Singapore) |

| AUABN | ABN (Australia) |

You can extend this by adding custom recognizers for local formats like Indonesian NIK, NPWP, and SIM.

Conclusion

Microsoft Presidio is an essential tool for every developer and data engineer working with sensitive data. With its combination of AI-powered NER and pattern matching, Presidio provides high flexibility for detecting and anonymizing various PII types.

Key takeaways:

  • Start simple: Use the basic analyzer and anonymizer first, then customize as needed
  • Build custom recognizers: For country-specific PII formats (Indonesian NIK, NPWP, etc.), create custom recognizers
  • Validate results: Always check for false positives and negatives, especially with non-English text
  • Use batch processing: For large datasets, leverage the batch analyzer and parallel processing
  • Audit trails: Log every anonymization operation for compliance requirements
  • Pipeline integration: Presidio integrates easily with FastAPI, Pandas, LangChain, and other tools
  • As data privacy regulations continue to tighten worldwide, the ability to automatically manage PII is becoming an invaluable skill. Presidio provides a robust foundation for building reliable and scalable data protection systems.

    Related Articles

    Semantic Router Tutorial: Smart Routing for LLM Applications

    Tutorial Semantic Router: Routing Cerdas untuk Aplikasi LLM Semantic Router adalah library Python yang memungkinkan Anda...

    BERTopic Tutorial: Modern Topic Modeling with Embeddings

    BERTopic: Pemodelan Topik Modern dengan Embedding BERTopic adalah library pemodelan topik yang menggabungkan embedding t...

    Sentence Transformers Tutorial: Embeddings, Similarity, and Rerankers

    Sentence Transformers: Embedding, Kemiripan Semantik, dan Reranker Sentence Transformers (sering disebut SBERT) adalah p...

    spaCy Tutorial: Industrial-Strength NLP in Python

    spaCy: NLP Kelas Industri di Python spaCy adalah pustaka open-source untuk pemrosesan bahasa alami (NLP) yang dirancang ...