[4Xin2] Poster session 2

[4Xin2-95] Evaluation of Prompts for Information Structure Recognition in Academic Documents

〇Hidekazu Nakawatase¹ (1.NII)

Keywords:Prompt, GAI, Academic information

The advancement of large language model technologies has enabled dialogue-based generative AIs to respond to natural language prompts. These AIs can execute tasks such as answering questions, recognizing document content, summarizing, and extracting information. However, the theory for designing optimal prompts is not yet established, and research has been conducted on effective prompt techniques. In this study, we verified the effectiveness of these techniques in a task of extracting cited references. Experiments utilized three Japanese academic papers with different layouts and bibliographic formats, employing ChatGPT (GPT-4). The results showed that for papers with a horizontal layout, simple prompts could effectively extract references. However, for vertically written documents with annotations in a two-column format, initial prompts were ineffective, and even revised prompts achieved only limited success. The improvement effect of prompt revision techniques based on prior research was observed only in certain techniques.