[2L3-J-9] Natural language processing, information retrieval: knowledge supply

[2L3-J-9-02] Monotonicity Dataset Creation on Crowdsourcing

Hitomi Yanaka^1,2, 〇Daisuke Bekki², Koji Mineshima², Satoshi Sekine¹, Kentaro Inui^3,1 (1. RIKEN Center for Advanced Intelligence Project, 2. Ochanomizu University, 3. Tohoku University)

Keywords:Natural Language Understanding, Reasoning Textual Entailment, Dataset Creation, Crowdsourcing

Large crowdsourced datasets are widely used for training and evaluating neural models on recognizing textual entailment (RTE). However, it is still unclear whether neural models can capture logical inferences, including monotonicity reasoning, for which no large naturalistic dataset has yet been developed. To investigate this issue, we introduce a method of creating a dataset for monotonicity reasoning by crowdsourcing and report the result of the first run. The error analysis indicates that workers tend to provide different answers from what logical entailment defines, for some downward monotonicity reasonings involving pragmatic reasoning.