JSAI2024

Presentation information

International Session

International Session » IS-1 Knowledge engineering

[2Q5-IS-1] Knowledge engineering

Wed. May 29, 2024 3:30 PM - 5:10 PM Room Q (Room 402)

Chair: Ryo Nishida (AIST)

4:50 PM - 5:10 PM

[2Q5-IS-1-05] Evaluating Color-Word Association in LLM

A Comparative Study of Human and AI

〇Makoto Fukushima1, Saki Kanada2, Shusuke Eshita2, Hiroshige Fukuhara2 (1. Deloitte Touche Tohmatsu LLC, 2. Sony Design Consulting Inc.)

Keywords:Large Language Model, Color, Design

Color is associated with various concepts, emphasizing functional significance in the design process. This study aims to evaluate the capability of Large Language Models (LLMs) in replicating human color-word associations. Leveraging a comprehensive dataset of human responses previously reported, with applications targeting color design [Fukushima 2021], we compare the predictive accuracy of LLMs against actual human associations between specific colors and words. We probed multiple LLMs with a series of multiple-choice questionnaires, originally designed for human participants. Our preliminary results indicate that LLMs achieve moderate success, with an accuracy rate of around 30-40% in predicting the best-voted words for all colors. We observed a marginal increase in performance for GPT-4, a multimodal LLM, compared to its predecessor, GPT-3.5. This suggests that while LLMs can mimic certain aspects of human cognitive processes, there are limitations in their ability to fully replicate human-level color-word associations. These limitations might stem from the inherent difficulties of symbol grounding in LLMs, or from a fundamentally different memory association structure in LLMs compared to humans.

Authentication for paper PDF access

A password is required to view paper PDFs. If you are a registered participant, please log on the site from Participant Log In.
You could view the PDF with entering the PDF viewing password bellow.

Password