[3S5-OS-7c] OS-7

[3S5-OS-7c-03] Evaluation of Treatment Progress Summaries Generated from Medical Treatment Records Using a Large Language Model

〇Kai Ishikawa¹, Yutaka Uno², Ryo Ishii³, Kunihiko Sadamasa¹, Daisaku Shibata², Masanori Tsujikawa², Atsuhiro Nakagawa⁴, Masafumi Oyamada¹, Masahiro Kubo², Yukio Katori³ (1. Data Science Laboratory, NEC Corporation, 2. Biometrics Research Laboratories, NEC Corporation, 3. Department of Otolaryngology-Head and Neck Surgery, Tohoku University Hospital, 4. Experience Design and Alliance Section, Tohoku University Hospital)

[[Online]]

Keywords:Treatment Progress Summaries , Medical Treatment Records , Large Language Model

It is urgent to improve the work efficiency of physicians because of the increase of their workload due to the declining birthrate and aging population, and a new overtime regulation that will be introduced in April 2024 as part of workstyle reform for physicians. We found that writing patient referral documents is a typical clerical work causing overtime work in the clinical observation, and developed a prototype system that supports physicians to create treatment progress summary using our Large Language Model. We evaluated the quality of draft summaries by the LLM and manually amended summaries with the help of ten physicians in the hospital. The averaged scores of draft summaries and amended summaries in ROUGE-1, ROUGE-2, and ROUGE-L were 46.6 and 42.9, 21.8 and 22.7, and 29.5 and 29.72, respectively. The draft summaries were natural and accurate according to physicians’ subjective evaluation. These results indicate that the proposed system has the potential to improve doctors’ work efficiency.