2:00 PM - 2:20 PM
[4J3-GS-6f-02] JSICK: Japanese Sentences Involving Compositional Knowledge Dataset
Keywords:Natural Language Inference, Recognizing Textual Entailment, Semantic Similarity, Dataset, Crowdsourcing
This paper introduces JSICK, a Japanese dataset for Recognizing Textual Entailment (RTE) and Semantic Textual Similarity (STS), manually translated from the English dataset SICK that focuses on compositional aspects of natural language inferences. Each sentence in JSICK is annotated with semantic tags to analyze whether models can capture diverse semantic phenomena. We perform a baseline evaluation of BERT-based RTE and STS models on JSICK, as well as a stress test in terms of word order scrambling in the JSICK test set. The results suggest that there is room for improving the performance on complex inferences and the generalization capacity of the models.
Authentication for paper PDF access
A password is required to view paper PDFs. If you are a registered participant, please log on the site from Participant Log In.
You could view the PDF with entering the PDF viewing password bellow.