10:20 AM - 10:40 AM
[4D1-GS-2-05] Enhanced Accuracy Estimation Method of Models in Production to Accelerate MLOps
Keywords:MLOps, Accuracy Estimation, Machine Learning
In MLOps, it is essential to estimate the performance of prediction models in production, especially when the target labels are not immediately available.
Existing methods utilize ``check'' models, which check the predictions of the prediction model, trained with the conventional cross-entropy loss.
This study presents a novel method to estimate accuracy using a check model based on our proposed GBCE loss, which strictly upper bounds the accuracy estimation error.
We analyze the generalization error bounds of both the proposed and the cross-entropy-based methods, thereby demonstrating the theoretical superiority of our method.
Through numerical experiments with multiple real-world data sets, we confirm the effectiveness of our proposed method, showing up to 56.3% reduction in accuracy estimation error.
Existing methods utilize ``check'' models, which check the predictions of the prediction model, trained with the conventional cross-entropy loss.
This study presents a novel method to estimate accuracy using a check model based on our proposed GBCE loss, which strictly upper bounds the accuracy estimation error.
We analyze the generalization error bounds of both the proposed and the cross-entropy-based methods, thereby demonstrating the theoretical superiority of our method.
Through numerical experiments with multiple real-world data sets, we confirm the effectiveness of our proposed method, showing up to 56.3% reduction in accuracy estimation error.
Authentication for paper PDF access
A password is required to view paper PDFs. If you are a registered participant, please log on the site from Participant Log In.
You could view the PDF with entering the PDF viewing password bellow.