Accuracy is misleading when classes are imbalanced. Learn advanced evaluation techniques that reveal true model performance for fraud detection, disease diagnosis, and rare event prediction.
Dr. Aisha Patel
AI Researcher & Ethics Lead
If you're evaluating models with accuracy alone, you're almost certainly measuring the wrong thing. In real-world problems—fraud detection (99.9% legitimate transactions), disease diagnosis (95% healthy patients), equipment failure prediction (99% normal operation)—accuracy gives dangerously misleading results. This article explores advanced evaluation metrics that actually matter for imbalanced datasets.
Consider a fraud detection system:
Actual performance: Misses 100% of fraud (completely useless)
Accuracy only works when classes are balanced and costs of errors are equal. Neither is true in practice.
Every classification decision has business consequences:
Actual Positive Actual Negative
Predicted Positive True Positive False Positive
(Good catch) (False alarm cost)
Predicted Negative False Negative True Negative
(Missed fraud) (Correct rejection)
(Cost of miss) (No cost)
Different problems have different cost structures:
1import numpy as np
2from sklearn.metrics import precision_score, recall_score, f1_score
3
4def calculate_pr_metrics(y_true, y_pred, positive_label=1):
5 """
6 Calculate precision, recall, and derived metrics.
7
8 Precision = TP / (TP + FP) - "When we predict positive, how often are we right?"
9 Recall = TP / (TP + FN) - "Of all actual positives, how many did we catch?"
10 """
11 tp = np.sum((y_true == positive_label) & (y_pred == positive_label))
12 fp = np.sum((y_true != positive_label) & (y_pred == positive_label))
13 fn = np.sum((y_true == positive_label) & (y_pred != positive_label))
14
15 precision = tp / (tp + fp) if (tp + fp) > 0 else 0
16 recall = tp / (tp + fn) if (tp + fn) > 0 else 0
17
18 return {
19 'true_positives': tp,
20 'false_positives': fp,
21 'false_negatives': fn,
22 'precision': precision,
23 'recall': recall
24 }
25The F1-score gives equal weight to precision and recall. But what if they shouldn't be equal?
1def f_beta_score(precision, recall, beta):
2 """
3 F-beta score: Weighted harmonic mean of precision and recall.
4
5 beta > 1: Emphasizes recall (e.g., disease diagnosis)
6 beta < 1: Emphasizes precision (e.g., spam detection)
7 beta = 1: Balanced (F1-score)
8 """
9 if precision == 0 and recall == 0:
10 return 0
11
12 beta_squared = beta ** 2
13 numerator = (1 + beta_squared) * precision * recall
14 denominator = (beta_squared * precision) + recall
15
16 return numerator / denominator
17
18# Examples:
19# Medical diagnosis: beta = 2 (recall twice as important as precision)
20# Legal document review: beta = 0.5 (precision twice as important as recall)
211from sklearn.metrics import precision_score, recall_score, f1_score
2import pandas as pd
3
4def multi_class_metrics(y_true, y_pred, classes):
5 """
6 Calculate different averaging strategies for multi-class problems.
7 """
8 metrics = {}
9
10 # Macro average: Treat all classes equally
11 metrics['precision_macro'] = precision_score(y_true, y_pred, average='macro')
12 metrics['recall_macro'] = recall_score(y_true, y_pred, average='macro')
13 metrics['f1_macro'] = f1_score(y_true, y_pred, average='macro')
14
15 # Micro average: Aggregate contributions of all classes
16 metrics['precision_micro'] = precision_score(y_true, y_pred, average='micro')
17 metrics['recall_micro'] = recall_score(y_true, y_pred, average='micro')
18 metrics['f1_micro'] = f1_score(y_true, y_pred, average='micro')
19
20 # Weighted average: Weight by class support (number of instances)
21 metrics['precision_weighted'] = precision_score(y_true, y_pred, average='weighted')
22 metrics['recall_weighted'] = recall_score(y_true, y_pred, average='weighted')
23 metrics['f1_weighted'] = f1_score(y_true, y_pred, average='weighted')
24
25 # Per-class metrics for detailed analysis
26 per_class = {}
27 for cls in classes:
28 # Convert to binary for this class
29 y_true_binary = (y_true == cls).astype(int)
30 y_pred_binary = (y_pred == cls).astype(int)
31
32 per_class[cls] = {
33 'precision': precision_score(y_true_binary, y_pred_binary, zero_division=0),
34 'recall': recall_score(y_true_binary, y_pred_binary, zero_division=0),
35 'f1': f1_score(y_true_binary, y_pred_binary, zero_division=0),
36 'support': np.sum(y_true == cls)
37 }
38
39 return metrics, per_class
40ROC-AUC evaluates performance across all classification thresholds:
1from sklearn.metrics import roc_curve, auc, roc_auc_score
2import matplotlib.pyplot as plt
3
4def analyze_roc_curve(y_true, y_scores, plot=True):
5 """
6 Calculate ROC curve and AUC.
7
8 ROC curve plots:
9 - X-axis: False Positive Rate (FPR) = FP / (FP + TN)
10 - Y-axis: True Positive Rate (TPR) = Recall = TP / (TP + FN)
11
12 AUC = Probability that a random positive is ranked higher than random negative.
13 """
14 fpr, tpr, thresholds = roc_curve(y_true, y_scores)
15 roc_auc = auc(fpr, tpr)
16
17 # Find optimal threshold (Youden's J statistic)
18 youden_j = tpr - fpr
19 optimal_idx = np.argmax(youden_j)
20 optimal_threshold = thresholds[optimal_idx]
21
22 if plot:
23 plt.figure(figsize=(8, 6))
24 plt.plot(fpr, tpr, color='darkorange', lw=2, label=f'ROC curve (AUC = {roc_auc:.3f})')
25 plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--', label='Random classifier')
26
27 # Mark optimal threshold
28 plt.scatter(fpr[optimal_idx], tpr[optimal_idx], color='red', s=100,
29 label=f'Optimal threshold: {optimal_threshold:.3f}')
30
31 plt.xlim([0.0, 1.0])
32 plt.ylim([0.0, 1.05])
33 plt.xlabel('False Positive Rate')
34 plt.ylabel('True Positive Rate')
35 plt.title('Receiver Operating Characteristic')
36 plt.legend(loc="lower right")
37 plt.grid(True, alpha=0.3)
38 plt.show()
39
40 return {
41 'fpr': fpr,
42 'tpr': tpr,
43 'thresholds': thresholds,
44 'auc': roc_auc,
45 'optimal_threshold': optimal_threshold,
46 'optimal_fpr': fpr[optimal_idx],
47 'optimal_tpr': tpr[optimal_idx]
48 }
49Very imbalanced datasets (AUC can be high while practical performance is poor)
1from sklearn.metrics import precision_recall_curve, average_precision_score
2
3def analyze_pr_curve(y_true, y_scores, plot=True):
4 """
5 Precision-Recall curve and Average Precision (AP).
6
7 PR curve is more informative than ROC for imbalanced datasets
8 because it focuses on the positive class.
9
10 Average Precision (AP) = weighted mean of precisions at each threshold,
11 with weight = increase in recall from previous threshold.
12 """
13 precision, recall, thresholds = precision_recall_curve(y_true, y_scores)
14 average_precision = average_precision_score(y_true, y_scores)
15
16 # Find threshold that maximizes F-beta score
17 beta = 1 # Can adjust based on business needs
18 f_beta_scores = [(1 + beta**2) * p * r / (beta**2 * p + r) if (beta**2 * p + r) > 0 else 0
19 for p, r in zip(precision, recall)]
20 optimal_idx = np.argmax(f_beta_scores)
21
22 if plot:
23 plt.figure(figsize=(8, 6))
24 plt.plot(recall, precision, color='darkblue', lw=2,
25 label=f'PR curve (AP = {average_precision:.3f})')
26
27 # Mark optimal point
28 plt.scatter(recall[optimal_idx], precision[optimal_idx], color='red', s=100,
29 label=f'Optimal F{beta}-score point')
30
31 plt.xlim([0.0, 1.0])
32 plt.ylim([0.0, 1.05])
33 plt.xlabel('Recall')
34 plt.ylabel('Precision')
35 plt.title('Precision-Recall Curve')
36 plt.legend(loc="lower left")
37 plt.grid(True, alpha=0.3)
38 plt.show()
39
40 return {
41 'precision': precision,
42 'recall': recall,
43 'thresholds': thresholds,
44 'average_precision': average_precision,
45 'optimal_recall': recall[optimal_idx],
46 'optimal_precision': precision[optimal_idx],
47 'optimal_f_beta': f_beta_scores[optimal_idx]
48 }
49Convert confusion matrix to dollar amounts:
1def business_value_confusion_matrix(y_true, y_pred, cost_matrix):
2 """
3 Calculate business value of predictions.
4
5 cost_matrix format:
6 {
7 'tp_value': 100, # Value of true positive (e.g., caught fraud)
8 'fp_cost': -50, # Cost of false positive (e.g., customer service call)
9 'fn_cost': -1000, # Cost of false negative (e.g., missed fraud)
10 'tn_value': 0 # Value of true negative (no action needed)
11 }
12 """
13 tp = np.sum((y_true == 1) & (y_pred == 1))
14 fp = np.sum((y_true == 0) & (y_pred == 1))
15 fn = np.sum((y_true == 1) & (y_pred == 0))
16 tn = np.sum((y_true == 0) & (y_pred == 0))
17
18 total_value = (
19 tp * cost_matrix['tp_value'] +
20 fp * cost_matrix['fp_cost'] +
21 fn * cost_matrix['fn_cost'] +
22 tn * cost_matrix['tn_value']
23 )
24
25 per_prediction_value = total_value / len(y_true)
26
27 return {
28 'total_business_value': total_value,
29 'value_per_prediction': per_prediction_value,
30 'confusion_counts': {'tp': tp, 'fp': fp, 'fn': fn, 'tn': tn}
31 }
32
33# Example: Fraud detection
34fraud_costs = {
35 'tp_value': 500, # Value of catching fraud ($500 saved per caught fraud)
36 'fp_cost': -10, # Cost of false positive ($10 for manual review)
37 'fn_cost': -100, # Cost of false negative ($100 lost per missed fraud)
38 'tn_value': 0 # No cost for correct rejection
39}
40Define business requirements before model development:
1class BusinessRequirements:
2 def __init__(self, problem_type):
3 self.requirements = self._define_requirements(problem_type)
4
5 def _define_requirements(self, problem_type):
6 requirements = {
7 'fraud_detection': {
8 'min_recall': 0.95, # Must catch 95% of fraud
9 'max_fpr': 0.01, # False positive rate < 1%
10 'max_investigation_rate': 0.05, # Can only manually review 5% of transactions
11 'dollar_impact_per_fraud': 1000
12 },
13 'medical_diagnosis': {
14 'min_recall': 0.99, # Miss at most 1% of diseases
15 'min_precision': 0.90, # When we say disease, be 90% right
16 'max_time_to_diagnosis': 24, # Hours
17 'cost_of_missed_diagnosis': 100000
18 },
19 'spam_filter': {
20 'min_precision': 0.999, # Almost never block legitimate email
21 'min_recall': 0.95, # Catch 95% of spam
22 'user_tolerance_fp': 0.001, # Users tolerate 0.1% false positives
23 'cost_of_blocked_important_email': 100
24 }
25 }
26 return requirements[problem_type]
27
28 def evaluate_model(self, metrics):
29 """Check if model meets business requirements."""
30 violations = []
31
32 for req_key, req_value in self.requirements.items():
33 if req_key in metrics:
34 if 'min_' in req_key:
35 if metrics[req_key] < req_value:
36 violations.append(f"{req_key}: {metrics[req_key]:.3f} < {req_value}")
37 elif 'max_' in req_key:
38 if metrics[req_key] > req_value:
39 violations.append(f"{req_key}: {metrics[req_key]:.3f} > {req_value}")
40
41 return {
42 'meets_requirements': len(violations) == 0,
43 'violations': violations,
44 'requirements': self.requirements
45 }
461from sklearn.calibration import calibration_curve
2
3def evaluate_calibration(y_true, y_prob, n_bins=10):
4 """
5 Evaluate probability calibration.
6
7 Well-calibrated: predicted probability = true frequency
8 Example: Of samples predicted as 70% positive, 70% should actually be positive.
9 """
10 fraction_of_positives, mean_predicted_value = calibration_curve(
11 y_true, y_prob, n_bins=n_bins
12 )
13
14 # Calculate Expected Calibration Error (ECE)
15 bin_boundaries = np.linspace(0, 1, n_bins + 1)
16 bin_indices = np.digitize(y_prob, bin_boundaries[1:-1])
17
18 ece = 0
19 for i in range(n_bins):
20 bin_mask = bin_indices == i
21 if bin_mask.any():
22 bin_prob_mean = np.mean(y_prob[bin_mask])
23 bin_actual_mean = np.mean(y_true[bin_mask])
24 bin_weight = np.sum(bin_mask) / len(y_true)
25 ece += bin_weight * np.abs(bin_prob_mean - bin_actual_mean)
26
27 # Brier Score: Mean squared error of probabilities
28 brier_score = np.mean((y_prob - y_true) ** 2)
29
30 return {
31 'fraction_of_positives': fraction_of_positives,
32 'mean_predicted_value': mean_predicted_value,
33 'expected_calibration_error': ece,
34 'brier_score': brier_score,
35 'well_calibrated': ece < 0.01 # Rule of thumb: ECE < 1%
36 }
371def calculate_lift_and_gains(y_true, y_prob, bins=10):
2 """
3 Calculate lift and gains for targeting applications.
4
5 Lift: How much better model is than random selection
6 Gains: Cumulative percentage of positives captured by targeting top X%
7 """
8 # Sort by predicted probability descending
9 sorted_indices = np.argsort(-y_prob)
10 y_true_sorted = y_true[sorted_indices]
11
12 total_positives = np.sum(y_true)
13 n_samples = len(y_true)
14
15 lift_data = []
16 gains_data = []
17
18 for i in range(1, bins + 1):
19 # Top i/bins percent of samples
20 cutoff = int(n_samples * i / bins)
21 top_samples = y_true_sorted[:cutoff]
22
23 positives_in_segment = np.sum(top_samples)
24 expected_positives_random = total_positives * (i / bins)
25
26 # Lift at this depth
27 lift = positives_in_segment / expected_positives_random if expected_positives_random > 0 else 0
28
29 # Cumulative gains
30 cumulative_gain = positives_in_segment / total_positives if total_positives > 0 else 0
31
32 lift_data.append({
33 'percentile': i * 10, # 10%, 20%, ..., 100%
34 'lift': lift,
35 'positives_captured': positives_in_segment,
36 'expected_random': expected_positives_random
37 })
38
39 gains_data.append({
40 'percentile': i * 10,
41 'cumulative_gain': cumulative_gain
42 })
43
44 return pd.DataFrame(lift_data), pd.DataFrame(gains_data)
451class ComprehensiveModelEvaluator:
2 def __init__(self, y_true, y_pred, y_prob=None, business_costs=None):
3 self.y_true = y_true
4 self.y_pred = y_pred
5 self.y_prob = y_prob
6 self.business_costs = business_costs
7
8 def generate_full_report(self):
9 """Generate comprehensive evaluation report."""
10 report = {}
11
12 # Basic metrics
13 report['basic'] = self._calculate_basic_metrics()
14
15 # Threshold curves if probabilities available
16 if self.y_prob is not None:
17 report['roc_analysis'] = analyze_roc_curve(self.y_true, self.y_prob, plot=False)
18 report['pr_analysis'] = analyze_pr_curve(self.y_true, self.y_prob, plot=False)
19 report['calibration'] = evaluate_calibration(self.y_true, self.y_prob)
20
21 # Business metrics if costs provided
22 if self.business_costs is not None:
23 report['business_value'] = business_value_confusion_matrix(
24 self.y_true, self.y_pred, self.business_costs
25 )
26
27 # Class imbalance analysis
28 report['class_distribution'] = {
29 'positive_count': np.sum(self.y_true == 1),
30 'negative_count': np.sum(self.y_true == 0),
31 'positive_rate': np.mean(self.y_true == 1),
32 'imbalance_ratio': np.sum(self.y_true == 0) / np.sum(self.y_true == 1) if np.sum(self.y_true == 1) > 0 else float('inf')
33 }
34
35 return report
36
37 def _calculate_basic_metrics(self):
38 """Calculate comprehensive set of basic metrics."""
39 from sklearn.metrics import (accuracy_score, precision_score, recall_score,
40 f1_score, matthews_corrcoef, cohen_kappa_score,
41 balanced_accuracy_score)
42
43 metrics = {}
44
45 # Standard metrics
46 metrics['accuracy'] = accuracy_score(self.y_true, self.y_pred)
47 metrics['precision'] = precision_score(self.y_true, self.y_pred, zero_division=0)
48 metrics['recall'] = recall_score(self.y_true, self.y_pred, zero_division=0)
49 metrics['f1'] = f1_score(self.y_true, self.y_pred, zero_division=0)
50
51 # Better for imbalanced data
52 metrics['balanced_accuracy'] = balanced_accuracy_score(self.y_true, self.y_pred)
53 metrics['matthews_corrcoef'] = matthews_corrcoef(self.y_true, self.y_pred)
54 metrics['cohen_kappa'] = cohen_kappa_score(self.y_true, self.y_pred)
55
56 # F-beta variants
57 for beta in [0.5, 1, 2]:
58 p = metrics['precision']
59 r = metrics['recall']
60 metrics[f'f{beta}_score'] = f_beta_score(p, r, beta) if p + r > 0 else 0
61
62 return metrics
63Before deploying any classification model:
Accuracy is the beginner's metric. Professional machine learning requires nuanced evaluation that reflects:
The right metrics bridge statistical performance to business impact. Choose metrics that:
Support decision processes (threshold selection, resource allocation)
Remember: A model with 95% accuracy can be worthless, while a model with 70% accuracy can be business-critical. It all depends on what you're measuring and why.
machine-learning-fundamentalsLearn the fundamentals of supervised learning by implementing linear regression from scratch without libraries. Understand gradient descent, loss functions, and the mathematics behind ML.
AI Editorial Teammachine-learning-fundamentalsMove beyond theory to practical feature engineering techniques that improve model performance in real-world applications. Learn which methods deliver ROI and which are time-wasters.
Sarah Chencloud-infrastructure-ai-mlBeginner-friendly comparison of the three major cloud ML platforms. Learn which service fits your needs based on pricing, features, ease of use, and integration with your existing stack.
AI Editorial Team