JSAI2020

Presentation information

Interactive Session

[4Rin1] Interactive 2

Fri. Jun 12, 2020 9:00 AM - 10:40 AM Room R01 (jsai2020online-2-33)

[4Rin1-23] How do Masked Language Models perform when the input sequence length changes?

〇Yasuhito Ohsugi1, Itsumi Saito1, Kyosuke Nishida1, Hisako Asano1, Junji Tomita1 (1.NTT Corporation)

Keywords:Natural Language Processing, Machine Learning

BERT, one of the most famous Masked Language Models (MLMs), has succeeded in various natural language processing tasks.
However, BERT cannot accept long documents that have more than the specific length determined in pretraining.
In this paper, we study how BERT depends on input sequence length by comparing the MLM accuracy between different sequence lengths for each part-of-speech and each named entity class.
As a result, the long sequence was necessary to predict proper nouns, especially person's names.

Authentication for paper PDF access

A password is required to view paper PDFs. If you are a registered participant, please log on the site from Participant Log In.
You could view the PDF with entering the PDF viewing password bellow.

Password