KDD 2020

View the Project on GitHub nplan-io/kdd2020-calibration

Tutorial outline

  1. Overview of calibration:
    • What is model calibration?
    • When to use calibration
    • Pitfalls of not considering calibration
  2. Measuring calibration (in Python):
    • Reliability diagrams
    • Expected calibration error
    • Maximum calibration error
    • RMS calibration error
  3. Literature review
    • Platt, J. - Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods
    • Niculescu-Mizil, Alexandru & Caruana, Rich. - Predicting good probabilities with supervised learning
    • Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. - On calibration of modern neural networks
  4. Methods for calibrating classifiers:
    • Isotonic regression
    • Platt scaling
    • Temperature scaling
    • Matrix and vector scaling
  5. How is calibration used in the real world?
First slide of the presentation How to Calibrate 
    your Neural Network Classifier: Getting True Probabilities from a Classification Model
Presentation slides