[4R1-OS-8a] OS-8

[4R1-OS-8a-01] The Multi-agent Fact-checking Based on Large Language Models

〇Yihan Dong¹, Takayuki Ito¹ (1. Kyoto Uniersity)

Keywords:multi-agent system, fact checking, Large Language Model

Fact-checking is a significant task since rumours and misinformation impact social networking services (SNS) and online discussions through misleading directions. Meanwhile, fact-checking with large language models (LLM) is becoming increasingly popular since the increase in the performance of LLM. However, the existing works have the following issues: 1) One single information source is assumed to be authoritative; 2) The judgement results made by LLM with the provided information are always considered to be completely credible; 3) Only the binary label classification task is insufficient due to the complexity of text on SNS.
Thus, we propose a framework called multi-agent fact-checking (MAFC) with multiple agents implemented and a unique mechanism to evaluate the credibility of the text. Our framework is tested through several comparative experiments. The first group's results prove that the proposed framework performs better than single agents and the multi-agent system with a simple voting mechanism in binary classification of rumours. Furthermore, The second group's results prove that it performs better than LLM in multiple-label classification. Finally, the challenges and obstacles existing in fact-checking fields, such as the definition standards and dataset creation, are discussed.