[3Xin2] Poster session 1

[3Xin2-72] Effectiveness of Knowledge Augmentation Prompts from Commonsense Knowledge Graphs

〇Kei Okada¹, Rafal Rzepka², Kenji Araki² (1.Graduate School of Information Science and Technology, Hokkaido University, 2.Faculty of Information Science and Technology, Hokkaido University)

Keywords:Commonsense knowledge graph, LLM

Before large language models (LLMs), models such as BERT were combined with external knowledge such as knowledge graphs to improve performance on various tasks. However, LLMs can achieve similar performance without the need to use external knowledge, by changing the input (prompt). In this study, we investigated the effectiveness of knowledge augmentation with knowledge graphs for LLMs. In the Japanese commonsense reasoning task JCommonsenseQA, we added knowledge from the commonsense knowledge graph ConceptNet to the input and tested five language models in mostly zero-shots settings. As validation data, we randomly selected 100 questions from the JCommonsenseQA train set. Then, we converted the ConceptNet knowledge into natural sentences, and extracted the top 10 most similar knowledge for each question. These 10 knowledge were added to the input prompt, and given to LLMs. Evaluation was done using exact match, and accuracy was calculated. Our experiment showed that the accuracy decreased or remained the same for all models when knowledge augmentation was applied. This suggests that knowledge augmentation prompt using commonsense knowledge graphs may not be effective in LLM.