[2-J-4-01] Preliminary Evaluation of Mapping Between In-House Drug Codes and Standard Drug Codes Using LLM with RAG
large language model, retrieval augmented generation, embedding vector
[Background]
Previous attempts to match drug name strings using vector similarity evaluation with word embedding have struggled to achieve high accuracy. Therefore, we plan to design a workflow that combines LLM and RAG. This workflow will utilise vector similarity to extract initial candidates, with the LLM further refining these selections. It is essential to determine the appropriate number of candidates for RAG; too many candidates hinder the LLM’s ability to narrow them down effectively, while too few may omit viable options.
[Methods]
We utilised the standard drug names from the MEDIS drug master and the uncoded in-house drug names submitted by three medical institutions. The names were vectorised using OpenAI's embedding model and stored in the FAISS vector database. Cosine similarity was used to assess the degree of approximation of the vectors, and the top five mapping candidates were extracted. A random sample of 5% of the mappings was evaluated for accuracy by one pharmacist.
[Results]
The results of the study revealed that 43 of the 300 sampled medicinal products were excluded due to their status as investigational drugs, medical devices, or other reasons making them unsuitable for evaluation. Among the remaining 257 products, 255 were accurately identified for mapping by the LLM. 10 candidates did not rank as the top 1.
[Discussion]
The findings of this study confirm that setting the Top K to 5 in the RAG can yield practical quality levels.
[Ethical consideration]
No patient samples or information were used in this study.
Previous attempts to match drug name strings using vector similarity evaluation with word embedding have struggled to achieve high accuracy. Therefore, we plan to design a workflow that combines LLM and RAG. This workflow will utilise vector similarity to extract initial candidates, with the LLM further refining these selections. It is essential to determine the appropriate number of candidates for RAG; too many candidates hinder the LLM’s ability to narrow them down effectively, while too few may omit viable options.
[Methods]
We utilised the standard drug names from the MEDIS drug master and the uncoded in-house drug names submitted by three medical institutions. The names were vectorised using OpenAI's embedding model and stored in the FAISS vector database. Cosine similarity was used to assess the degree of approximation of the vectors, and the top five mapping candidates were extracted. A random sample of 5% of the mappings was evaluated for accuracy by one pharmacist.
[Results]
The results of the study revealed that 43 of the 300 sampled medicinal products were excluded due to their status as investigational drugs, medical devices, or other reasons making them unsuitable for evaluation. Among the remaining 257 products, 255 were accurately identified for mapping by the LLM. 10 candidates did not rank as the top 1.
[Discussion]
The findings of this study confirm that setting the Top K to 5 in the RAG can yield practical quality levels.
[Ethical consideration]
No patient samples or information were used in this study.