[1E5-GS-9] Natural language processing, information retrieval: Estimate and analysis

[1E5-GS-9-03] Analyzing Difficulty of Machine Comprehension focusing on Examples Answerable by BERT without Question

〇Hongyu Li¹, Tengyang Chen¹, Takehito Utsuro¹, Yasuhide Kawada² (1. University of Tsukuba, 2. Logworks Co., Ltd.)

Keywords:question answering, machine comprehension, answerable, BERT, difficulty

In the field of machine comprehension (MC), the task of an MC model is to predict the answer (A) from a
question (Q) and related context (C) of the question. However, in this paper, it is discovered that there exist
examples that can be correctly answered by an MC model BERT where only the context of the example is given
without the question being given, which means that the difficulty of examples of machine comprehension vary.
Based on this finding, this paper proposes a method based on BERT which splits the training examples of the MC
dataset SQuAD1.1 into “easy to answer” and “hard to answer” ones. Experimental evaluation results of comparing
the two models, one of which is trained with the “easy to answer” examples only, while the other of which is trained
with the “hard to answer” examples only, show that the latter outperforms the former.