Tutorial Lengkap Finetuning EasyOCR untuk Dataset Custom
EasyOCR adalah library OCR (Optical Character Recognition) open-source yang powerful dan mendukung lebih dari 80 bahasa. Namun, untuk kasus penggunaan spesifik seperti font khusus, dokumen historis, handwriting, atau format dokumen yang unik, finetuning model EasyOCR bisa meningkatkan akurasi secara signifikan.
Dalam tutorial ini, kita akan mempelajari cara melakukan finetuning EasyOCR dari awal hingga evaluasi model.
Prasyarat
Sebelum memulai, pastikan Anda memiliki:
- Python 3.7+ (disarankan Python 3.8 atau 3.9)
- GPU dengan CUDA support (sangat direkomendasikan, minimal 8GB VRAM)
- Minimal 16GB RAM sistem
- Dataset gambar teks dengan label ground truth (minimal 1000 samples untuk hasil baik)
- Disk space minimal 10GB untuk model dan dataset
Instalasi dan Setup Environment
1. Clone Repository EasyOCR
# Clone repository
git clone https://github.com/JaidedAI/EasyOCR.git
cd EasyOCR
Checkout ke branch stable (opsional)
git checkout v1.7.0
2. Setup Virtual Environment
# Buat virtual environment
python -m venv easyocrenv
Aktivasi
Untuk Linux/Mac:
source easyocrenv/bin/activate
Untuk Windows:
easyocrenv\Scripts\activate
3. Install Dependencies
# Install PyTorch dengan CUDA support
Sesuaikan versi CUDA dengan sistem Anda
pip install torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/cu118
Install requirements EasyOCR
pip install -r requirements.txt
Install dependencies tambahan untuk training
pip install tensorboard
pip install lmdb
pip install pillow
pip install opencv-python
pip install albumentations
pip install python-Levenshtein
4. Verifikasi Instalasi
import torch
print(f"PyTorch version: {torch.version}")
print(f"CUDA available: {torch.cuda.isavailable()}")
print(f"CUDA version: {torch.version.cuda}")
print(f"GPU count: {torch.cuda.devicecount()}")
if torch.cuda.isavailable():
print(f"GPU name: {torch.cuda.getdevicename(0)}")
Persiapan Dataset
Dataset adalah komponen paling penting dalam finetuning. Kualitas dan kuantitas dataset sangat mempengaruhi hasil akhir.
1. Struktur Folder Dataset
Buat struktur folder seperti ini:
dataset/
├── raw/
│ ├── train/
│ │ ├── img001.jpg
│ │ ├── img002.jpg
│ │ └── ...
│ ├── validation/
│ │ ├── img001.jpg
│ │ └── ...
│ └── test/
│ ├── img001.jpg
│ └── ...
├── labels/
│ ├── trainlabels.txt
│ ├── vallabels.txt
│ └── testlabels.txt
└── lmdb/
├── train/
└── validation/
2. Format Label File
File label menggunakan format TSV (Tab-Separated Values):
trainlabels.txt:img001.jpg Hello World
img
002.jpg Invoice #12345
img003.jpg Total: $1,250.00
img004.jpg PT Rubythalib Data Konsulta
Setiap baris berisi:
- Nama file gambar
- Tab character (\t)
- Teks ground truth
- Gambar harus jelas dan readable
- Minimal resolusi 64x256 pixels
- Variasi font, ukuran, dan style
- Berbagai kondisi pencahayaan
- Include noise dan distorsi realistis
- Balanced character distribution
3. Script Preprocessing Dataset
Buat file preparedataset.py:
import os
import cv2
import numpy as np
from PIL import Image
from pathlib import Path
def preprocessimage(imagepath, outputpath):
"""
Preprocessing gambar untuk training OCR
"""
# Baca gambar
img = cv2.imread(str(imagepath))
if img is None:
print(f"Error reading {imagepath}")
return False
# Convert ke grayscale
gray = cv2.cvtColor(img, cv2.COLORBGR2GRAY)
# Noise reduction
denoised = cv2.fastNlMeansDenoising(gray, None, 10, 7, 21)
# Adaptive thresholding (opsional, tergantung dataset)
# thresh = cv2.adaptiveThreshold(denoised, 255,
# cv2.ADAPTIVETHRESHGAUSSIANC,
# cv2.THRESHBINARY, 11, 2)
# Resize jika terlalu besar (max height 128)
height, width = denoised.shape
if height > 128:
scale = 128 / height
newwidth = int(width scale)
denoised = cv2.resize(denoised, (newwidth, 128))
# Save
cv2.imwrite(str(outputpath), denoised)
return True
def validatedataset(labelfile, imagedir):
"""
Validasi dataset - cek semua gambar ada dan readable
"""
issues = []
with open(labelfile, 'r', encoding='utf-8') as f:
lines = f.readlines()
print(f"Validating {len(lines)} samples...")
for idx, line in enumerate(lines):
parts = line.strip().split('\t')
if len(parts) != 2:
issues.append(f"Line {idx+1}: Invalid format (expected 2 columns)")
continue
imagename, text = parts
imagepath = os.path.join(imagedir, imagename)
# Check file exists
if not os.path.exists(imagepath):
issues.append(f"Line {idx+1}: Image not found - {imagename}")
continue
# Check image readable
img = cv2.imread(imagepath)
if img is None:
issues.append(f"Line {idx+1}: Cannot read image - {imagename}")
continue
# Check text not empty
if len(text.strip()) == 0:
issues.append(f"Line {idx+1}: Empty text label - {imagename}")
if issues:
print(f"\n❌ Found {len(issues)} issues:")
for issue in issues[:10]: # Show first 10
print(f" - {issue}")
if len(issues) > 10:
print(f" ... and {len(issues)-10} more")
return False
else:
print(f"✅ All {len(lines)} samples validated successfully!")
return True
Jalankan validasi
if name == "main":
# Validasi training set
validatedataset(
"dataset/labels/trainlabels.txt",
"dataset/raw/train"
)
# Validasi validation set
validatedataset(
"dataset/labels/vallabels.txt",
"dataset/raw/validation"
)
Jalankan validasi:
python preparedataset.py
4. Konversi Dataset ke Format LMDB
EasyOCR menggunakan LMDB (Lightning Memory-Mapped Database) untuk training yang lebih efisien. Buat file createlmdbdataset.py:
import os
import lmdb
import cv2
import numpy as np
from tqdm import tqdm
def createlmdbdataset(imagedir, labelfile, outputpath):
"""
Membuat LMDB dataset untuk EasyOCR training
Args:
imagedir: Directory berisi gambar
labelfile: Path ke file label (TSV format)
outputpath: Path output LMDB database
"""
# Baca label file
with open(labelfile, 'r', encoding='utf-8') as f:
lines = [line.strip() for line in f.readlines()]
# Calculate map size (1TB)
mapsize = 1099511627776
# Create LMDB environment
env = lmdb.open(outputpath, mapsize=mapsize)
cache = {}
cnt = 1
print(f"Processing {len(lines)} samples...")
for idx, line in enumerate(tqdm(lines)):
# Parse line
parts = line.split('\t')
if len(parts) != 2:
print(f"Skipping line {idx+1}: Invalid format")
continue
imagename, label = parts
imagepath = os.path.join(imagedir, imagename)
# Check file exists
if not os.path.exists(imagepath):
print(f"Skipping line {idx+1}: Image not found - {imagename}")
continue
# Read image as binary
with open(imagepath, 'rb') as f:
imagebin = f.read()
if not imagebin:
print(f"Skipping line {idx+1}: Empty image - {imagename}")
continue
# Create keys (format yang dibutuhkan EasyOCR)
imagekey = f'image-{cnt:09d}'.encode()
labelkey = f'label-{cnt:09d}'.encode()
# Store in cache
cache[imagekey] = imagebin
cache[labelkey] = label.encode('utf-8')
cnt += 1
# Write to LMDB in batches (setiap 1000 samples)
if cnt % 1000 == 0:
with env.begin(write=True) as txn:
for key, value in cache.items():
txn.put(key, value)
cache = {}
print(f"Written {cnt-1} samples to LMDB")
# Write remaining cache
if cache:
with env.begin(write=True) as txn:
for key, value in cache.items():
txn.put(key, value)
# Write total number of samples
with env.begin(write=True) as txn:
txn.put('num-samples'.encode(), str(cnt-1).encode())
env.close()
print(f"\n✅ Created LMDB dataset with {cnt-1} samples at {outputpath}")
return cnt-1
def verifylmdb(lmdbpath):
"""
Verifikasi LMDB dataset
"""
env = lmdb.open(lmdbpath, readonly=True, lock=False)
with env.begin() as txn:
numsamples = txn.get('num-samples'.encode())
if numsamples:
numsamples = int(numsamples.decode())
print(f"Total samples in LMDB: {numsamples}")
# Check beberapa sample
print("\nChecking first 3 samples:")
for i in range(1, min(4, numsamples+1)):
imagekey = f'image-{i:09d}'.encode()
labelkey = f'label-{i:09d}'.encode()
imagebin = txn.get(imagekey)
label = txn.get(labelkey)
if imagebin and label:
print(f" Sample {i}: Label = '{label.decode('utf-8')}', Image size = {len(imagebin)} bytes")
else:
print(f" Sample {i}: Missing data!")
env.close()
if name == "main":
# Create training LMDB
print("Creating training LMDB...")
traincount = createlmdbdataset(
imagedir="dataset/raw/train",
labelfile="dataset/labels/trainlabels.txt",
outputpath="dataset/lmdb/train"
)
print("\nVerifying training LMDB...")
verifylmdb("dataset/lmdb/train")
print("\n" + "="50 + "\n")
# Create validation LMDB
print("Creating validation LMDB...")
valcount = createlmdbdataset(
imagedir="dataset/raw/validation",
labelfile="dataset/labels/vallabels.txt",
outputpath="dataset/lmdb/validation"
)
print("\nVerifying validation LMDB...")
verifylmdb("dataset/lmdb/validation")
print("\n" + "="50)
print(f"✅ Dataset ready for training!")
print(f" Training samples: {traincount}")
print(f" Validation samples: {valcount}")
print(f" Ratio: {valcount/traincount100:.1f}% validation")
Jalankan konversi:
python createlmdbdataset.py
Output yang diharapkan:
Processing 5000 samples...
100%|████████████████████| 5000/5000 [00:15<00:00, 320.45it/s]
✅ Created LMDB dataset with 5000 samples at dataset/lmdb/train
Verifying training LMDB...
Total samples in LMDB: 5000
Checking first 3 samples:
Sample 1: Label = 'Hello World', Image size = 45231 bytes
Sample 2: Label = 'Invoice #12345', Image size = 38942 bytes
Sample 3: Label = 'Total: $1,250.00', Image size = 41523 bytes
Konfigurasi Training
1. Memahami Arsitektur Model EasyOCR
EasyOCR menggunakan arsitektur 4-stage:
2. File Konfigurasi Training
Buat file trainconfig.yaml:
# Data configuration
data:
traindata: "./dataset/lmdb/train"
validdata: "./dataset/lmdb/validation"
selectdata: ["train"]
batchratio: [1.0]
totaldatausageratio: 1.0 # 1.0 = use all data
batchmaxlength: 25 # max text length
imgH: 64 # image height
imgW: 256 # image width
rgb: False # False = grayscale, True = RGB
character: "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!\"#$%&'()+,-./:;<=>?@[\\]^{|}~ " # character set
batchsize: 192
numworkers: 4
Model architecture
model:
inputchannel: 1 # 1 for grayscale, 3 for RGB
outputchannel: 512
hiddensize: 256
Transformation: "TPS" # None | TPS
FeatureExtraction: "ResNet" # VGG | RCNN | ResNet
SequenceModeling: "BiLSTM" # None | BiLSTM
Prediction: "Attn" # CTC | Attn
Training configuration
training:
numiter: 300000 # total training iterations
batchsize: 192
valInterval: 2000 # validation setiap 2000 iterations
savedmodel: "./savedmodels" # directory untuk save checkpoints
FT: False # False = train from scratch, True = finetune
continuemodel: "" # path to pretrained model (if FT=True)
adam: False # False = Adadelta, True = Adam
lr: 1.0 # learning rate
beta1: 0.9 # for Adam
rho: 0.95 # for Adadelta
eps: 1e-8
gradclip: 5 # gradient clipping
numgpu: 1 # number of GPUs
Augmentation
augmentation:
useaugmentation: True
rotation: 5 # random rotation degree
scalefactor: 0.1 # random scale
blurprob: 0.3
noiseprob: 0.3
3. Character Set Configuration
PENTING: Sesuaikan character set dengan kebutuhan dataset Anda.Contoh untuk berbagai kasus:
Angka saja (digit recognition):character = "0123456789"
Alphanumeric + basic punctuation:
character = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!.,- "
Indonesian text (dengan huruf khusus):
character = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ àáèéìíòóùú"
Full ASCII:
import string
character = string.printable[:-5] # exclude \t\n\r\v\f
Tips Character Set:
- Include SEMUA karakter yang muncul di dataset
- Jangan include karakter yang tidak ada di dataset
- Case sensitive: 'A' != 'a'
- Space character penting untuk text dengan spasi
4. Script Analisis Character Distribution
Untuk memastikan character set sudah benar, buat analyzechars.py:
from collections import Counter
import matplotlib.pyplot as plt
def analyzecharacterdistribution(labelfile):
"""
Analisis distribusi karakter dalam dataset
"""
allchars = []
with open(labelfile, 'r', encoding='utf-8') as f:
for line in f:
parts = line.strip().split('\t')
if len(parts) == 2:
text = parts[1]
allchars.extend(list(text))
# Count characters
charcounts = Counter(allchars)
print(f"Total characters: {len(allchars)}")
print(f"Unique characters: {len(charcounts)}")
print("\nTop 20 most common characters:")
for char, count in charcounts.mostcommon(20):
if char == ' ':
char = ''
elif char == '\t':
char = ''
elif char == '\n':
char = ''
print(f" '{char}': {count} ({count/len(allchars)100:.2f}%)")
# All unique characters
print("\nAll unique characters:")
uniquechars = sorted(charcounts.keys())
print(''.join(uniquechars))
# Generate character string for config
print("\nCharacter string untuk config:")
print(f'character = "{' '.join(uniquechars)}"')
return charcounts
if name == "main":
print("Analyzing training set...")
trainchars = analyzecharacterdistribution("dataset/labels/trainlabels.txt")
print("\n" + "="60 + "\n")
print("Analyzing validation set...")
valchars = analyzecharacterdistribution("dataset/labels/vallabels.txt")
# Check if val set has chars not in train set
trainset = set(trainchars.keys())
valset = set(valchars.keys())
extrainval = valset - trainset
if extrainval:
print(f"\n⚠️ Warning: Validation set has {len(extrainval)} characters not in training set:")
print(f" {extrainval}")
Jalankan analisis:
python analyzechars.py
Proses Training
1. Training dari Scratch
Training model baru dari nol (tanpa pretrained weights):
python train.py \
--traindata dataset/lmdb/train \
--validdata dataset/lmdb/validation \
--selectdata train \
--batchratio 1.0 \
--Transformation TPS \
--FeatureExtraction ResNet \
--SequenceModeling BiLSTM \
--Prediction Attn \
--batchsize 192 \
--numiter 300000 \
--valInterval 2000 \
--savedmodel ./savedmodels/ \
--lr 1.0 \
--workers 4 \
--manualSeed 1111 \
--rgb False
Penjelasan parameter:
- --traindata
: Path ke LMDB training dataset - --validdata
: Path ke LMDB validation dataset - --Transformation TPS
: Gunakan Thin Plate Spline untuk handle distortion - --FeatureExtraction ResNet
: Gunakan ResNet untuk feature extraction - --SequenceModeling BiLSTM
: Bidirectional LSTM untuk sequence modeling - --Prediction Attn
: Attention mechanism untuk prediction - --batchsize 192
: Batch size (reduce jika OOM) - --numiter 300000
: Total training iterations - --valInterval 2000
: Validasi setiap 2000 iterations - --lr 1.0
: Learning rate (untuk Adadelta) - --workers 4
: Jumlah data loader workers - --rgb False
: Grayscale images
2. Finetuning dari Pretrained Model
Finetuning dari model pretrained EasyOCR untuk accelerate training:
Download pretrained model:# Latin alphabet model (English)
wget https://github.com/JaidedAI/EasyOCR/releases/download/v1.3/lating2.zip
unzip lating2.zip
Atau model bahasa lain
Indonesian
wget https://github.com/JaidedAI/EasyOCR/releases/download/v1.3/indonesiang2.zip
Finetune command:
python train.py \
--traindata dataset/lmdb/train \
--validdata dataset/lmdb/validation \
--selectdata train \
--batchratio 1.0 \
--Transformation TPS \
--FeatureExtraction ResNet \
--SequenceModeling BiLSTM \
--Prediction Attn \
--batchsize 192 \
--numiter 50000 \
--valInterval 1000 \
--savedmodel ./finetunedmodels/ \
--lr 0.0001 \
--workers 4 \
--manualSeed 1111 \
--savedmodel lating2.pth \
--FT
Perbedaan penting saat finetuning:
- --savedmodel lating2.pth
: Path ke pretrained model - --FT
: Flag untuk enable finetuning mode - --lr 0.0001
: Learning rate LEBIH KECIL (1000x lebih kecil) - --numiter 50000
: Iterasi LEBIH SEDIKIT (6x lebih sedikit)
3. Monitoring Training dengan TensorBoard
EasyOCR otomatis log metrics ke TensorBoard. Buka di terminal baru:
tensorboard --logdir=./savedmodels/
Buka browser: http://localhost:6006
- Good: Smooth decrease
- Bad: Plateau atau naik
- Solution jika bad: Reduce learning rate
- Monitor overfitting: jika train acc naik tapi val acc turun
- Solution: Early stopping, regularization, atau augmentation
- Good: < 0.05 (5% error)
- Acceptable: 0.05 - 0.15
- Bad: > 0.15
4. Training Script dengan Automatic Checkpointing
Buat wrapper script trainwithmonitor.py:
import subprocess
import os
import sys
from datetime import datetime
def trainwithmonitoring():
"""
Training dengan automatic monitoring dan checkpoint management
"""
# Training configuration
config = {
'traindata': 'dataset/lmdb/train',
'validdata': 'dataset/lmdb/validation',
'batchsize': 192,
'numiter': 300000,
'valInterval': 2000,
'savedmodel': './savedmodels/',
'Transformation': 'TPS',
'FeatureExtraction': 'ResNet',
'SequenceModeling': 'BiLSTM',
'Prediction': 'Attn',
'lr': 1.0,
'workers': 4,
}
# Create save directory dengan timestamp
timestamp = datetime.now().strftime('%Y%m%d%H%M%S')
savedir = f"experiments/exp{timestamp}"
os.makedirs(savedir, existok=True)
config['savedmodel'] = savedir
# Build command
cmd = ['python', 'train.py']
for key, value in config.items():
cmd.extend([f'--{key}', str(value)])
# Log configuration
with open(f'{savedir}/config.txt', 'w') as f:
f.write(f"Training started: {timestamp}\n")
f.write(f"Command: {' '.join(cmd)}\n\n")
f.write("Configuration:\n")
for key, value in config.items():
f.write(f" {key}: {value}\n")
print(f"Starting training...")
print(f"Logs will be saved to: {savedir}")
print(f"Monitor with: tensorboard --logdir={savedir}")
# Run training
try:
process = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universalnewlines=True
)
# Stream output
logfile = f'{savedir}/training.log'
with open(logfile, 'w') as f:
for line in process.stdout:
print(line, end='') # Print to console
f.write(line) # Write to log file
f.flush()
process.wait()
if process.returncode == 0:
print(f"\n✅ Training completed successfully!")
else:
print(f"\n❌ Training failed with code {process.returncode}")
except KeyboardInterrupt:
print("\n⚠️ Training interrupted by user")
process.terminate()
process.wait()
except Exception as e:
print(f"\n❌ Error during training: {e}")
if name == "main":
trainwithmonitoring()
Jalankan:
python trainwithmonitor.py
Testing dan Evaluasi Model
1. Testing Model dengan EasyOCR Reader
Setelah training selesai, test model dengan script testmodel.py:
import easyocr
import cv2
import os
from pathlib import Path
def testcustommodel(modelpath, testimagepath):
"""
Test custom trained model
"""
# Initialize reader dengan custom model
reader = easyocr.Reader(
['en'], # Language (sesuaikan)
gpu=True,
modelstoragedirectory='./models/',
usernetworkdirectory=modelpath,
recognetwork='custom'
)
# Read image
result = reader.readtext(testimagepath, detail=1)
# Print results
print(f"\nResults for: {testimagepath}")
print("-" 60)
for bbox, text, confidence in result:
print(f"Text: {text}")
print(f"Confidence: {confidence:.4f}")
print(f"BBox: {bbox}")
print()
return result
def visualizeresults(imagepath, result, outputpath):
"""
Visualisasi hasil OCR pada gambar
"""
img = cv2.imread(imagepath)
for bbox, text, conf in result:
# Extract coordinates
topleft = tuple(map(int, bbox[0]))
bottomright = tuple(map(int, bbox[2]))
# Draw rectangle
color = (0, 255, 0) if conf > 0.8 else (0, 165, 255) # Green if confident, Orange if not
cv2.rectangle(img, topleft, bottomright, color, 2)
# Draw text
cv2.putText(
img,
f"{text} ({conf:.2f})",
(topleft[0], topleft[1] - 10),
cv2.FONTHERSHEYSIMPLEX,
0.6,
color,
2
)
# Save result
cv2.imwrite(outputpath, img)
print(f"Visualization saved to: {outputpath}")
Test pada single image
if name == "main":
modelpath = "./savedmodels/bestaccuracy.pth"
testimage = "dataset/test/sample001.jpg"
result = testcustommodel(modelpath, testimage)
visualizeresults(testimage, result, "outputvisualized.jpg")
2. Evaluasi Comprehensive pada Test Set
Buat script evaluatemodel.py untuk evaluasi menyeluruh:
import easyocr
import cv2
import os
from tqdm import tqdm
import Levenshtein
import numpy as np
import json
def calculatemetrics(predicted, groundtruth):
"""
Calculate CER, WER, dan exact match accuracy
"""
# Character Error Rate
cer = Levenshtein.distance(predicted, groundtruth) / max(len(groundtruth), 1)
# Word Error Rate
predwords = predicted.split()
gtwords = groundtruth.split()
wer = Levenshtein.distance(' '.join(predwords), ' '.join(gtwords)) / max(len(gtwords), 1)
# Exact match
exactmatch = 1.0 if predicted == groundtruth else 0.0
return {
'cer': cer,
'wer': wer,
'exactmatch': exactmatch
}
def evaluateondataset(modelpath, testdatadir, labelfile):
"""
Evaluasi model pada test dataset
"""
# Initialize reader
print("Loading model...")
reader = easyocr.Reader(
['en'],
gpu=True,
modelstoragedirectory='./models/',
usernetworkdirectory=modelpath,
recognetwork='custom'
)
# Load test labels
with open(labelfile, 'r', encoding='utf-8') as f:
testsamples = [line.strip().split('\t') for line in f.readlines()]
results = []
totalcer = 0
totalwer = 0
totalexact = 0
errors = []
print(f"Evaluating on {len(testsamples)} samples...")
for imgname, groundtruth in tqdm(testsamples):
imgpath = os.path.join(testdatadir, imgname)
if not os.path.exists(imgpath):
print(f"Warning: Image not found - {imgpath}")
continue
# Predict
try:
result = reader.readtext(imgpath, detail=0)
predicted = ' '.join(result) if result else ""
except Exception as e:
print(f"Error processing {imgname}: {e}")
predicted = ""
# Calculate metrics
metrics = calculatemetrics(predicted, groundtruth)
# Store results
results.append({
'image': imgname,
'groundtruth': groundtruth,
'predicted': predicted,
*metrics
})
totalcer += metrics['cer']
totalwer += metrics['wer']
totalexact += metrics['exactmatch']
# Track significant errors
if metrics['cer'] > 0.3: # CER > 30%
errors.append({
'image': imgname,
'groundtruth': groundtruth,
'predicted': predicted,
'cer': metrics['cer']
})
# Calculate averages
nsamples = len(results)
avgmetrics = {
'avgcer': totalcer / nsamples,
'avgwer': totalwer / nsamples,
'exactmatchaccuracy': totalexact / nsamples,
'totalsamples': nsamples
}
# Print summary
print("\n" + "="60)
print("EVALUATION RESULTS")
print("="60)
print(f"Total samples: {nsamples}")
print(f"Average CER: {avgmetrics['avgcer']:.4f} ({avgmetrics['avgcer']100:.2f}%)")
print(f"Average WER: {avgmetrics['avgwer']:.4f} ({avgmetrics['avgwer']100:.2f}%)")
print(f"Exact Match Accuracy: {avgmetrics['exactmatchaccuracy']:.4f} ({avgmetrics['exactmatchaccuracy']100:.2f}%)")
print(f"Significant errors (CER > 30%): {len(errors)}")
# CER Distribution
cervalues = [r['cer'] for r in results]
print("\nCER Distribution:")
print(f" Min: {min(cervalues):.4f}")
print(f" Max: {max(cervalues):.4f}")
print(f" Median: {np.median(cervalues):.4f}")
print(f" Std Dev: {np.std(cervalues):.4f}")
# Show worst errors
if errors:
print("\nTop 5 worst predictions:")
errors.sort(key=lambda x: x['cer'], reverse=True)
for i, err in enumerate(errors[:5], 1):
print(f"\n{i}. {err['image']} (CER: {err['cer']:.4f})")
print(f" GT: {err['groundtruth']}")
print(f" Pred: {err['predicted']}")
# Save detailed results
with open('evaluationresults.json', 'w', encoding='utf-8') as f:
json.dump({
'summary': avgmetrics,
'detailedresults': results,
'errors': errors
}, f, indent=2, ensureascii=False)
print(f"\n✅ Detailed results saved to: evaluationresults.json")
return avgmetrics, results
if name == "main":
modelpath = "./savedmodels/bestaccuracy.pth"
testdatadir = "dataset/raw/test"
labelfile = "dataset/labels/testlabels.txt"
evaluateondataset(modelpath, testdatadir, labelfile)
Jalankan evaluasi:
python evaluatemodel.py
Output expected:
Loading model...
Evaluating on 1000 samples...
100%|████████████████████| 1000/1000 [02:15<00:00, 7.38it/s]
============================================================
EVALUATION RESULTS
============================================================
Total samples: 1000
Average CER: 0.0342 (3.42%)
Average WER: 0.0891 (8.91%)
Exact Match Accuracy: 0.8520 (85.20%)
Significant errors (CER > 30%): 15
CER Distribution:
Min: 0.0000
Max: 0.8571
Median: 0.0200
Std Dev: 0.0876
Top 5 worst predictions:
img789.jpg (CER: 0.8571)
GT: Invoice #INV-2024-001
Pred: Jnvoice #1NV-Z0Z4-00l
✅ Detailed results saved to: evaluationresults.json
Optimasi dan Tips Advanced
1. Data Augmentation untuk Improve Generalization
Implementasi augmentation yang lebih aggressive:
import albumentations as A
import cv2
def gettrainingaugmentation():
"""
Augmentation pipeline untuk training
"""
return A.Compose([
# Geometric transforms
A.Rotate(limit=10, p=0.5, bordermode=cv2.BORDERCONSTANT, value=255),
A.ShiftScaleRotate(
shiftlimit=0.1,
scalelimit=0.15,
rotatelimit=10,
bordermode=cv2.BORDERCONSTANT,
value=255,
p=0.5
),
# Noise
A.OneOf([
A.GaussNoise(varlimit=(10.0, 50.0), p=1.0),
A.MultiplicativeNoise(multiplier=(0.9, 1.1), p=1.0),
], p=0.3),
# Blur
A.OneOf([
A.GaussianBlur(blurlimit=(3, 7), p=1.0),
A.MotionBlur(blurlimit=5, p=1.0),
], p=0.3),
# Brightness & Contrast
A.RandomBrightnessContrast(
brightnesslimit=0.3,
contrastlimit=0.3,
p=0.5
),
# Additional
A.CLAHE(cliplimit=2.0, tilegridsize=(8, 8), p=0.3),
A.Sharpen(alpha=(0.2, 0.5), lightness=(0.5, 1.0), p=0.3),
])
Apply augmentation
transform = gettrainingaugmentation()
image = cv2.imread('sample.jpg')
augmented = transform(image=image)['image']