[4J3-GS-6f] 言語メディア処理：データセットとその利用

[4J3-GS-6f-02] JSICK: Japanese Sentences Involving Compositional Knowledge Dataset

〇Hitomi Yanaka¹, Koji Mineshima² (1. Riken, 2. Keio University)

Keywords:Natural Language Inference, Recognizing Textual Entailment, Semantic Similarity, Dataset, Crowdsourcing

This paper introduces JSICK, a Japanese dataset for Recognizing Textual Entailment (RTE) and Semantic Textual Similarity (STS), manually translated from the English dataset SICK that focuses on compositional aspects of natural language inferences. Each sentence in JSICK is annotated with semantic tags to analyze whether models can capture diverse semantic phenomena. We perform a baseline evaluation of BERT-based RTE and STS models on JSICK, as well as a stress test in terms of word order scrambling in the JSICK test set. The results suggest that there is room for improving the performance on complex inferences and the generalization capacity of the models.