[2Q4-IS-5] Human interface, education aid

[2Q4-IS-5-05] Impact of Role Prompting on Automated Essay Scoring Using GPT Models

〇Chenhui Li¹, Lui Yoshida¹ (1. The University of Tokyo)

Keywords:Automated Essay Scoring, Large Language Models, Role Prompting, GPT (Generative Pre-trained Transformer)

Recent advancements in generative AI, particularly in Automated Essay Scoring (AES), have shown great potential, yet their accuracy remains insufficient compared to existing methods. The aim of this study is to explore the impact of role prompting on improving the performance of Large Language Models (LLMs) in AES tasks. In this research, we analyzed 240 essays written by non-native English speakers, extracted from eight prompts of the TOEFL11 corpus. Using each three versions of GPT-3.5 and GPT-4, essays were scored employing prompts representing seven different roles, and the results were evaluated against human ratings using Quadratic Weighted Kappa (QWK). The findings indicate that roles presumed to be advantageous did not necessarily enhance AES performance. Moreover, the gpt-4-0613 model demonstrated the highest effectiveness. This study contributes to the ongoing discussion on optimizing LLMs for AES, providing insights into their potential and limitations.