[1O4-OS-18a] OS-18

[1O4-OS-18a-04] Evaluation of Entity Linking Accuracy Using LLMs on Pre-Diagnostic Medical History Texts

〇Takashi Nishibayashi¹, Mitsuhisa Ota¹, Masahiro Kazama¹ (1. Ubie, Inc.)

Keywords:NLP, healthcare, knowledge base

Entity linking is essential for converting medical texts into structured data for medical information systems. In this study, we evaluated the accuracy of a large language model (LLM) in extracting medical entities from pre-diagnostic patient history texts and mapping them to a structured knowledge base. We specifically focused on symptoms and symptom attributes, including severity, onset, progression, and presentation style. We conducted entity linking experiments with a Long-Context LLM with few-shot examples using a medical knowledge base comprising approximately 4,000 concepts. Evaluation on a dataset of 130 annotated clinical scenarios yielded an F1 score of 0.8 for the entities representing the binary presence of symptoms and 0.7 for the scale entities representing attributes such as severity. The results suggest that LLM-assisted entity linking can support but not fully automate information extraction. Thought process outputs improved performance, while ambiguity in the knowledge base negatively impacted accuracy. Future work includes refining knowledge representation for better LLM integration.