[3Win5] Poster session 3

[3Win5-12] Adversarial Benchmark for Evaluating Stereotypes in Japanese Culture

〇Taihei Shiotani¹, Masahiro Kaneko^2,1, Ayana Niwa², Yuki Maruyama¹, Daisuke Oba¹, Masanari Ohi¹, Naoaki Okazaki^1,3,4 (1.Institute of Science Tokyo, 2.MBZUAI, 3.AIST, 4.NII LLMC)

Keywords:NLP, LLM, Fairness

In bias evaluation of large language models (LLMs), non-English-speaking regions often rely on translated English datasets. However, such translated datasets are based on Western cultural norms and fail to fully reflect the ethical values and social norms of different cultural contexts. In this study, we construct an adversarial benchmark, JUBAKU, designed to evaluate bias specific to Japanese culture. We manually create dialogue data to elicit biases in LLMs and assess nine Japanese LLMs using JUBAKU. The results show that all models performed worse than the random baseline, revealing their vulnerability to biases unique to Japanese culture.