[3A1-GS-10] AI application:

[3A1-GS-10-03] Research on Agent Evaluation Methods in LLM-based Multi-Agent Systems

〇Takaaki Tanaka¹, Shun Otsubo², Yuki Hayashi², Ito Kotaro², Takuya Hatakeyama¹, Takashi Yoshida¹, Nobuyuki Ishikawa¹ (1. Hakuhodo Technologies Inc., 2. NTT DATA Mathematical Systems Inc.)

Keywords:AI Agent, Large Language Models, Multi Agent, LLM Agent, Human-AI Interaction

Multi-agent systems leveraging Large Language Models (LLMs) represent a paradigm in which multiple AI agents collaborate or compete to accomplish complex tasks. These systems have been explored for a wide range of applications, including improving the accuracy of question-answering, simulating real-world interactions, and enhancing the efficiency of software development. However, methods for evaluating the effectiveness of individual agents within multi-agent systems remain underexplored. In this study, we utilized an LLM-based multi-agent system we developed, called Nomatica, to investigate and evaluate methods for assessing the effectiveness of individual agents during tasks such as free discussions, ideation, and review sessions. The evaluation particularly focused on the utility of RAG (Retrieval Augmented Generation) agents, which leverage RAG technology. The results demonstrated that utilizing RAG agents and task-specific agents contributes to improving the overall system performance, providing valuable insights for the development of agent evaluation methodologies.