Model Evaluation Metrics
Choose the metric that matches the cost of mistakes, not just the default.
Regression
| Metric | Meaning |
|---|---|
| MAE | Average absolute error; robust, easy to explain |
| MSE / RMSE | Penalises large errors more; RMSE has the target's units |
| R² | Share of variance explained (1 is perfect) |
Classification: the confusion matrix
Predicted + Predicted -
Actual + TP FN
Actual - FP TN
| Metric | Formula | Use when |
|---|---|---|
| Accuracy | (TP + TN) / total | Classes are balanced |
| Precision | TP / (TP + FP) | False alarms are costly (spam filter) |
| Recall (sensitivity) | TP / (TP + FN) | Misses are costly (disease screening) |
| F1 | 2PR / (P + R) | Balance precision and recall |
| ROC-AUC | Area under TPR vs FPR curve | Ranking quality across thresholds |
Why accuracy can mislead
If 99% of transactions are legitimate, a model that always predicts "legitimate" has 99% accuracy but catches no fraud. Use precision, recall, F1 or PR-AUC on imbalanced data.
from sklearn.metrics import classification_report, confusion_matrix
pred = clf.predict(X_test)
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred))
Splitting strategies
| Strategy | When |
|---|---|
| Train / validation / test | Plenty of data |
| k-fold cross-validation | Limited data |
| Stratified split | Imbalanced classes |
| Time-based split | Time series (never shuffle across time) |
Diagnosing a model
| Symptom | Likely cause | Try |
|---|---|---|
| High train and test error | Underfitting | More features, a more flexible model |
| Low train, high test error | Overfitting | More data, regularisation, simpler model |
| Good offline, bad in production | Data drift or leakage | Monitor inputs, fix the pipeline |
Loading Comments...