Choosing and calculating evaluation metrics

A model's score means little until you know which metric was used and why. Screenings often give a confusion matrix and ask you to calculate a metric, or describe a business problem and ask which metric fits.

The confusion matrix

For a yes/no classifier, every prediction falls into one of four cells:

TP = 40   (predicted yes, actually yes)
FP = 10   (predicted yes, actually no)
FN = 20   (predicted no, actually yes)
TN = 930  (predicted no, actually no)

Classification metrics

• Accuracy = (TP + TN) / total = 970 / 1000 = 0.97.
• Precision = TP / (TP + FP) = 40 / 50 = 0.80. Of the cases flagged yes, how many were right?
• Recall = TP / (TP + FN) = 40 / 60 = 0.667. Of the real yes cases, how many did we catch?
• F1 = 2TP / (2TP + FP + FN) = 80 / 110 = 0.727. This is the harmonic mean of precision and recall, not the simple average (0.733).

Accuracy looks excellent here, yet a third of real positives were missed. On imbalanced data, a model that always predicts the majority class can score high accuracy with zero recall.

Matching the metric to the cost

• Missing a case is costly (disease screening, fraud): focus on recall.
• False alarms are costly (blocking a genuine payment, flagging good emails as spam): focus on precision.
• Both matter and classes are imbalanced: use F1, and report precision and recall alongside it.

Changing the decision threshold moves precision and recall in opposite directions, so state the threshold you used.

Regression metrics

For errors 3, -1, 1, -3 and 0:

• MAE (mean absolute error) = (3 + 1 + 1 + 3 + 0) / 5 = 1.6.
• MSE (mean squared error) = (9 + 1 + 1 + 9 + 0) / 5 = 4.
• RMSE = square root of MSE = 2.

RMSE is in the same units as the target and punishes large errors more than MAE does. If a few big misses are very costly, RMSE is a better guide.

Habit to build: whenever you quote a metric, add one sentence on what it means in the business problem, for example "recall 0.67 means we catch two out of three fraud cases".