[4-F-3-03] Mitigating Context Loss in Clinical Notes Auto-Summarization through Information Theory-Based Dynamic Decoding
Large Language Model, Electronic Medical Records, Auto-summarization, Information Theory
Purpose
When using LLMs achieve Clinical Notes Auto-summarization, accuracy will drop drastically after input exceeding Context Window Size.I refer to this problem as context loss caused by long inputs. Our Goal is solve this problem to generate human-like summary automately.
Methods
To solve this problem, we applied Native Bayes Context Extend and Information Theory for optimizing decoding layer of LLMs, enables the model to select the appropriate sentence from the entire Clinical Notes to generate human-like summary.Our dataset is extracted from the clinical notes of doctors and nurses. 1000 patients are randomly selected as training set and 100 as test set.To evaluate the performance we employed the commonly used metrics called ROUGE-L. If the ROUGE-L score is higher, it means that the prediction result is more similar to the human. For example, Dave Van Veen et al applied various generative AI for clinical document summarization and compared by them using ROUGE-L score. In their research ChatGPT-3.5 applied for Open-i dataset is up to 0.35.
Results
Average ROUGE-L score of our model was 0.13. Although the result is approx. three times smaller than the Van Veen’s result, it is difficult to discuss about goodness of the model because the nature of dataset applied to them is far different and our model size is 20 times smaller than them. We need to make more detailed analysis to clarify the effect of our approach.
Conclusion
We applied Native Bayes Context Extend coupled with slight modification to overcome the context loss caused by long inputs.
Ethical considerations
This study was approved by Ethics Committee of Kyoto University Graduate School and Faculty of Medicine as No. R4333.
When using LLMs achieve Clinical Notes Auto-summarization, accuracy will drop drastically after input exceeding Context Window Size.I refer to this problem as context loss caused by long inputs. Our Goal is solve this problem to generate human-like summary automately.
Methods
To solve this problem, we applied Native Bayes Context Extend and Information Theory for optimizing decoding layer of LLMs, enables the model to select the appropriate sentence from the entire Clinical Notes to generate human-like summary.Our dataset is extracted from the clinical notes of doctors and nurses. 1000 patients are randomly selected as training set and 100 as test set.To evaluate the performance we employed the commonly used metrics called ROUGE-L. If the ROUGE-L score is higher, it means that the prediction result is more similar to the human. For example, Dave Van Veen et al applied various generative AI for clinical document summarization and compared by them using ROUGE-L score. In their research ChatGPT-3.5 applied for Open-i dataset is up to 0.35.
Results
Average ROUGE-L score of our model was 0.13. Although the result is approx. three times smaller than the Van Veen’s result, it is difficult to discuss about goodness of the model because the nature of dataset applied to them is far different and our model size is 20 times smaller than them. We need to make more detailed analysis to clarify the effect of our approach.
Conclusion
We applied Native Bayes Context Extend coupled with slight modification to overcome the context loss caused by long inputs.
Ethical considerations
This study was approved by Ethics Committee of Kyoto University Graduate School and Faculty of Medicine as No. R4333.