[2Q5-IS-1] Knowledge engineering

[2Q5-IS-1-05] Evaluating Color-Word Association in LLM

A Comparative Study of Human and AI

〇Makoto Fukushima¹, Saki Kanada², Shusuke Eshita², Hiroshige Fukuhara² (1. Deloitte Touche Tohmatsu LLC, 2. Sony Design Consulting Inc.)

Keywords:Large Language Model, Color, Design

Color is associated with various concepts, emphasizing functional significance in the design process. This study aims to evaluate the capability of Large Language Models (LLMs) in replicating human color-word associations. Leveraging a comprehensive dataset of human responses previously reported, with applications targeting color design [Fukushima 2021], we compare the predictive accuracy of LLMs against actual human associations between specific colors and words. We probed multiple LLMs with a series of multiple-choice questionnaires, originally designed for human participants. Our preliminary results indicate that LLMs achieve moderate success, with an accuracy rate of around 30-40% in predicting the best-voted words for all colors. We observed a marginal increase in performance for GPT-4, a multimodal LLM, compared to its predecessor, GPT-3.5. This suggests that while LLMs can mimic certain aspects of human cognitive processes, there are limitations in their ability to fully replicate human-level color-word associations. These limitations might stem from the inherent difficulties of symbol grounding in LLMs, or from a fundamentally different memory association structure in LLMs compared to humans.