[4Rin1] Interactive Session 2

[4Rin1-06] Corpus Filtering Focusing on Named Entities for Neural Machine Translation

〇Hiroki Homma¹, Hayahide Yamagishi¹, Yukio Matsumura¹, Mamoru Komachi¹ (1. Tokyo Metropolitan University)

Keywords:Machine translation, Corpus filtering

Some parallel corpora include sentences that disturb learning of machine translation systems. By removing such noisy sentences like containing many out-of-vocabulary from the training corpus, it is expected to makes better translations. In this paper, we focus on the sentences containing named entities because most of the named entities fall into out-of-vocabulary due to low-frequencies. We propose two kinds of filtering methods, using byte pair encoding and using named entity recognition. By removing noisy sentences from a training corpus on Japanese-English language pair, BLEU scores improve statistically significantly by 0.5 points in both proposed methods. Analysis revealed that both our methods overcome the mistakes such as suffix of the noun, determiner, and sentence lengths.