Tutorial outline

1. Overview of calibration:
   • What is model calibration?
   • When to use calibration
   • Pitfalls of not considering calibration
2. Measuring calibration (in Python):
   • Reliability diagrams
   • Expected calibration error
   • Maximum calibration error
   • RMS calibration error
3. Literature review
   • Platt, J. - Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods
   • Niculescu-Mizil, Alexandru & Caruana, Rich. - Predicting good probabilities with supervised learning
   • Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. - On calibration of modern neural networks
4. Methods for calibrating classifiers:
   • Isotonic regression
   • Platt scaling
   • Temperature scaling
   • Matrix and vector scaling
5. How is calibration used in the real world?

Presentation slides