TY  - JOUR
AU  - Gruber, Sebastian
AU  - Bach, Francis
TI  - Optimizing Estimators of Squared Calibration Errors in Classification
JO  - Transactions on machine learning research
VL  - nn
SN  - 2835-8856
CY  - [Amherst, Massachusetts]
PB  - OpenReview.net
M1  - DKFZ-2025-00760
SP  - nn
PY  - 2025
N1  - epub
AB  - In this work, we propose a mean-squared error-based risk that enables the comparison and optimization of estimators of squared calibration errors in practical settings. Improving the calibration of classifiers is crucial for enhancing the trustworthiness and interpretability of machine learning models, especially in sensitive decision-making scenarios. Although various calibration (error) estimators exist in the current literature, there is a lack of guidance on selecting the appropriate estimator and tuning its hyperparameters. By leveraging the bilinear structure of squared calibration errors, we reformulate calibration estimation as a regression problem with independent and identically distributed (i.i.d.) input pairs. This reformulation allows us to quantify the performance of different estimators even for the most challenging calibration criterion, known as canonical calibration. Our approach advocates for a training-validation-testing pipeline when estimating a calibration error on an evaluation dataset. We demonstrate the effectiveness of our pipeline by optimizing existing calibration estimators and comparing them with novel kernel ridge regression-based estimators on standard image classification tasks.
KW  - Machine Learning (cs.LG) (Other)
KW  - Machine Learning (stat.ML) (Other)
KW  - FOS: Computer and information sciences (Other)
LB  - PUB:(DE-HGF)16
DO  - DOI:DOI:10.48550/arXiv.2410.07014
UR  - https://inrepo02.dkfz.de/record/300314
ER  -