[1B5-OS-41c] OS-41

[1B5-OS-41c-04] JDERW: Japanese LLM Deduction benchmark requiring a world model

〇TAISEI OZAkI¹, Takumi Matsushita², Tsuyoshi Miura³, Shohei Taniguchi³, Yutaka Matsuo³ (1. Osaka Metropolitan University, 2. Institute of Science Tokyo, 3. The University of Tokyo)

Keywords:World Model, Deduction, benchmark

Recent studies suggest that Large Language Models (LLMs) demonstrate capabilities beyond simple next-token prediction, leading to discussions about their potential acquisition of world models. This paper introduces Basic-JDERW, a deductive reasoning benchmark dataset that requires fundamental world understanding. The dataset comprises 103 QA tasks that necessitate the application of basic world models, ranging from physical phenomena comprehension to common sense reasoning and action planning, categorized into six types: causal reasoning, temporal reasoning, spatial reasoning, abstract concept reasoning, common sense reasoning, and planning. Through evaluation experiments with eight LLMs, we analyzed model performance across categories and examined correlations with existing benchmarks. Notably, llama3.3-70B-instruct demonstrated superior performance in categories requiring physical understanding, such as temporal and spatial reasoning. This research offers new perspectives on evaluating basic world understanding capabilities glimpsed through LLMs' reasoning abilities and aims to contribute to understanding the relationship between linguistic reasoning and world comprehension capabilities.