[3F4-OS-42a] OS-42

[3F4-OS-42a-03] Fairness Evaluation of Large Language Models Using Psychological Methods

〇Junya Suzuki¹, Makoto Fukushima² (1. Deloitte Tohmatsu Cyber LLC, 2. Deloitte Touche Tohmatsu LLC)

Keywords:AI, Large Language Model, AI Governance, Fairness Evaluation, Psychological

Ensuring fairness in large language models (LLM) is one of the challenges in AI governance. The purpose of this paper is to utilize the characteristics of LLM discovered using psychological methods in existing research, and to find the possibility of a new index of fairness. One of the characteristics of LLM is that when instructions imitating a specific gender or race are given to LLM, unexpected differences occur in the percentage of correct answers to specific questions based on the instructions. By using this characteristic, we show the possibility of using it as an index to measure hidden stereotypes inherent in LLM. As another characteristic, higher STICSA scores (“anxious ”) were associated with a higher proportion of LLM biased responses. Based on this relationship, we show that the STICSA score can be used as a bias evaluation index for various inputs. As a conclusion of this paper, we discuss the significance of applying these psychological characteristics of LLM as a fairness evaluation index in AI governance and find its possibility.