[3Xin2] Poster session 1

[3Xin2-26] Acquiring Bidirectionality via Large and Small Language Model

〇Takumi Goto^1,2, Hiroyoshi Nagao¹, Yuta Koreeda¹ (1.Research & Development Group, Hitachi, Ltd., 2.NARA Institute of Science and Technology)

Keywords:Natural Language Processing, Large Language Model, Named Entity Recognition

In this study, we raise the issue of uni-directionality when applying large causal language models to classical NLP tasks. As a solution, we propose a method of utilizing the concatenated representations of a newly trained small-scale backward language model as input for downstream tasks. Through experiments in named entity recognition tasks, we demonstrate that introducing backward model improves the benchmark performance more than 10 points. Furthermore, we report that the proposed method is especially effective for rare domains and in few-shot learning settings.