[3F1-GS-10] AI application: Language model

[3F1-GS-10-03] Calibrating Biases of Large Language Model for Recommendation

〇Yusuke Kumagae¹, Ikumi Ito², Go Kamoda², Sho Yokoi^2,3 (1. Hakuhodo DY Holdings Inc., 2. Tohoku University, 3. RIKEN)

Keywords:Large Language Model, Recommendation, Calibration, bias

Large Language Models (LLMs) are expected to solve the challenges current recommendation systems face. We investigate biases that arise when using LLMs as recommendation systems and verify the validity of existing calibration methods. We first demonstrate the rating bias caused by using LLMs in a few-shot manner on actual data. We then apply several calibration methods from other classification tasks, such as sentiment analysis and natural language inference, to mitigate this bias. Our findings reveal these methods to be insufficient for recommendation tasks. We further question the assumptions made by the existing methods and discuss strategies for improvement.