[3P1-OS-46a] OS-46

[3P1-OS-46a-04] Assessing Logical Inference Capabilities of Large Language Models Through RDF Schema Entailment Rules

〇Taichi Hosokawa¹, Takeshi Morita¹ (1. Aoyama Gakuin University)

Keywords:RDF Schema entailment rules, counterfactual knowledge, large language models, ontology, logical inference capability

Large language models (LLMs) have recently demonstrated remarkable performance across various language tasks. However, they continue to face significant challenges in logical inference, often depending on pre-trained knowledge rather than engaging in genuine inference processes. Furthermore, their ability to perform inference tasks within ontology-based frameworks remains underexamined. This study focuses on RDF Schema entailment rules, leveraging two types of knowledge datasets: real-world datasets constructed from Linked Open Data and counterfactual datasets created by systematically modifying real-world knowledge from multiple perspectives. We propose a novel evaluation methodology to assess the inference capabilities of LLMs using these datasets. In the experiments, LLMs were provided with prompts containing rules and knowledge to generate inference outputs. These outputs were evaluated using real-world and counterfactual knowledge based on precision, recall, and F1 scores. The findings revealed inference failures in rare knowledge structures and a reliance on resource name patterns, underscoring the limitations of LLMs in inference with RDF Schema entailment rules.