[3K3-J-2] Machine learning: analysis and validations of models

[3K3-J-2-03] Do the AUC and log-loss evaluate CTR prediction models properly?

〇Satoshi KATAGIRI¹ (1. F@N Communications, Inc.)

Keywords:Machine Learning, CTR Prediction, Evaluation Metrics, Calibration, Online Advertising

Click-through rate (CTR) prediction is one of the most important task for web advertising platform companies. However, CTR prediction is a non-standard machine learning task, so conventional metrics, for example, area under the Receiver Operating Characteristic curve (AUC), and log-loss, a.k.a. cross-entropy, and so on, can be improper. Our target is develop a new metrices for CTR prediction. In this article, we state the drawbacks of such conventional metrics and perspective of a metric based on the calibration plot approach.