[2Win5] Poster session 2

[2Win5-69] Investigation of LLM-based Agent Adaptation in Financial Domain through User Feedback

〇Shunsuke Onuma¹ (1.The Japan Research Institute, Limited)

Keywords:LLM, Agent, Industorial Application

LLM-based agent applications for autonomous decision-making and action are gaining attention. To implement agents in specialized tasks, developers need to incorporate domain-specific knowledge into them. However, it is challenging to organize necessary information including implicit knowledge in advance. To address this, one of the possible approaches is providing agents with domain-specific knowledge through user feedback during task execution. In this paper, we present results of adapting an agent to a financial task through user feedback. Using the Certified Skilled Professional of Financial Planning Exam for our evaluation, we provided feedback to the agent and reviewed changes of the exam scores, reasoning process and behavior. We observed the agent's behavior that did not utilized the feedback. Besides the behavior changes coming from the nature of LLM models, we identified points to improve the feedback method, the agent's utilization of feedback and the tools available to the agent.