[3Rin4] Interactive 1

[3Rin4-76] Fine-tuning of BERT Based on Triplet Network with Relations between Wikipedia Articles

〇Hayahide Yamagishi¹, Takafumi Suzuki¹, Seiya Inagi¹ (1.Fuji Xerox Co., Ltd.)

Keywords:Natural Language Processing, Sentence Embeddings, Fine-tuning

Recently, sentence embedding approaches by fine-tuning a pre-trained language model have achieved significant success in several NLP tasks. In order to improve the semantic representation of sentences, Ein Dor et al. have introduced a triplet network to fine-tune a pre-trained model, where each triplet consists of an anchor text and its related and unrelated sentences extracted from Wikipedia articles. In this work, we propose to improve the triplet network to learn the relatedness of articles by using the hyperlink network in Wikipedia. We first quantify the relatedness of two articles with the number of hops in the hyperlink network. The triplets are rebuilt according to the relatedness, and moreover, the loss function is modified to incorporate such relatedness to the model. Our result shows that our proposed model improves the performance in tasks of thematic similarity prediction.