[4Xin2] Poster session 2

[4Xin2-57] Performance Evaluation of Large Language Models on Negation Understanding using DanSto Dataset

Arisa Abe¹, 〇Rafal Rzepka², Kenji Araki² (1.Graduate School of Information Science and Technology, Hokkaido University, 2.Faculty of Information Science and Technology, Hokkaido University)

Keywords:NLP, LLM, negation understanding

This research aims to investigate the performance of large language models by conducting experiments using a Japanese story dataset. For this purpose, we utilized recently developed DanSto dataset consisting of approximately 9,000 sets containing 5-sentence stories. Various tests were conducted, such as swapping affirmations and negations in the concluding sentences and creating new endings, to assess whether large language models can correctly choose the appropriate final sentence. The results of this study revealed that some current large language models face difficulties in understanding negations in stories written from scratch in Japanese language. Future challenges include experiments on causality for story comprehension and the automatic expansion of the DanSto dataset using generative models.