Model Evaluation

Use this skill for rigorously assessing model performance, comparing alternatives, and diagnosing issues.

When to use this skill

• •Model training complete — need performance assessment
• •Comparing multiple models/algorithms
• •Diagnosing overfitting/underfitting
• •Hyperparameter tuning
• •Production readiness check

Evaluation workflow

1. •

   Cross-validation strategy

   • •K-fold (default for most cases)
   • •Stratified K-fold (classification with imbalance)
   • •TimeSeriesSplit (temporal data)
   • •GroupKFold (grouped/clustered data)
2. •

   Choose appropriate metrics

   • •Classification: accuracy, precision, recall, F1, ROC-AUC, PR-AUC
   • •Regression: MAE, RMSE, R², MAPE
   • •Ranking: NDCG, MAP
   • •Business: custom metrics tied to outcomes
3. •

   Analyze performance

   • •Cross-validation mean ± std
   • •Validation curve (bias-variance tradeoff)
   • •Learning curves (data sufficiency)
   • •Error analysis by segment
4. •

   Model comparison

   • •Statistical significance (paired t-test, McNemar)
   • •Calibration (for probability outputs)
   • •Speed vs accuracy tradeoffs

Quick tool selection

Core implementation rules

1) Always use proper validation

from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=cv, scoring='roc_auc')
print(f"CV AUC: {scores.mean():.3f} (+/- {scores.std() * 2:.3f})")

2) Match metrics to problem

# Classification with imbalance
from sklearn.metrics import classification_report, confusion_matrix

print(classification_report(y_true, y_pred))
# Focus on F1, precision/recall for minority class

# Regression
from sklearn.metrics import mean_absolute_error, root_mean_squared_error

print(f"MAE: {mean_absolute_error(y_true, y_pred):.3f}")
print(f"RMSE: {root_mean_squared_error(y_true, y_pred):.3f}")

3) Analyze errors systematically

# Error by segment
errors = y_pred != y_true
error_df = X_test[errors]
error_df['true'] = y_true[errors]
error_df['pred'] = y_pred[errors]

# Analyze patterns in errors
print(error_df.groupby('category').size())

4) Track experiments

import mlflow

with mlflow.start_run():
    mlflow.log_params(params)
    mlflow.log_metrics({'auc': auc, 'f1': f1})
    mlflow.sklearn.log_model(model, 'model')

Common anti-patterns

• •❌ Single train/test split without CV
• •❌ Optimizing wrong metric (accuracy on imbalanced data)
• •❌ Data leakage in preprocessing
• •❌ Not checking calibration for probability outputs
• •❌ Ignoring inference speed/memory constraints
• •❌ No error analysis or debugging bad predictions

Progressive disclosure

• •../references/cross-validation.md — CV strategies for different data types
• •../references/metrics-guide.md — Choosing and interpreting metrics
• •../references/hyperparameter-tuning.md — Optuna, Ray Tune patterns
• •../references/experiment-tracking.md — MLflow, W&B setup

Related skills

• •@data-science-feature-engineering — Features to evaluate
• •@data-engineering-orchestration — Production model deployment
• •@data-engineering-observability — Model monitoring in production

References

• •sklearn Model Selection
• •sklearn Metrics
• •Optuna Documentation
• •MLflow Tracking