[2H5-GS-11] AI and Society:

[2H5-GS-11-02] Simulating a specialist using LLM

a case study on modeling a law professor

〇Makoto Fukushima¹, Yuki Yamamoto², Takafumi Ochiai³, Tatsuhiko Inatani⁴ (1. Deloitte Touche Tohmatsu LLC, Deloitte Analytics R&D, 2. Deloitte Tohmatsu Risk Advisory LLC, 3. Atsumi & Sakai, Policy Reserach Institute, 4. Kyoto University, Graduate School of Law)

Keywords:LLM, Law, Big Five personality traits

Large Language Models (LLMs) continue to improve in their conversational performance, and their ability to replicate human behavior is advancing. With the increase in the number of tokens available for in-context learning in LLMs, it has become easier to configure LLMs to act as experts. However, clear procedures for replicating the intellectual activities of highly skilled experts using LLMs have not yet been established. This study aims to establish procedures for configuring expert LLMs by constructing an LLM that replicates the behavior of a law professor. System prompts were created based on the professor's own written works and feedback from acquaintances, and the behavior of the LLM incorporating these prompts was compared to the professor's actual responses. The LLM reproduced the professor’s answers from the General Social Survey with approximately 60% accuracy, while the replication of personality traits remained at a lower level, showing little correlation with the traits reported by the law professor. On the other hand, the LLM’s personality showed a higher correlation with those obtained from the professor’s acquaintances' ratings. Considering the gap between self-assessment and external evaluation of personality, as well as the biases present in the LLM's responses, we discuss methods for future performance improvements.