[1B4-GS-2] Machine learning: Expression learning

[1B4-GS-2-01] Automated Model Performance Evaluation via Contrastive Learning of Distilled Surrogate Model

〇Makoto Kawano¹, Kazuki Kawamura¹ (1. The University of Tokyo)

Keywords:Unlabeled Performance Evaluation, Knowledge Distillation, Contrastive Learning

Real-world machine learning system operation suffers from performance degradation due to data distribution shift, which occurs during operation and leads to lower accuracy compared to model validation. Detecting this performance degradation enables appropriate measures such as model retraining or structural revision. However, continuous labeling of operational data is not realistic due to the high cost. Therefore, this study focuses on estimating the performance of a model on unlabeled test data. Since direct calculation of accuracy on test data is impossible without labels, previous studies have attempted to estimate test accuracy using distances or metrics correlated with it. One such study utilizes adversarial accuracy, but it requires simultaneous adversarial training with the model to be evaluated, rendering it inapplicable to pre-trained models. To address this, we propose CoLDS, a method that estimates the test performance of any model without labels by converting the model to be evaluated into a surrogate model using knowledge distillation and performing adversarial training on the surrogate model. This paper evaluates the effectiveness of CoLDS through experiments and reports the results.