[2B4-GS-6] Language media processing: language model

[2B4-GS-6-04] A Study on Building a Japanese General-Purpose Language Model for Temporal Common Sense Understanding

〇Hikari Funabiki¹, Mayuko Kimura¹, Lis Kanashiro Pereira¹, Ichiro Kobayashi¹ (1. Ochanomizu University)

Keywords:Temporal Commonsense

In order to understand events expressed in natural language, it is important to understand time. However, since they are often omitted from descriptions, it is necessary to have common sense knowledge about various temporal aspects of events. Therefore, we aim to build a Japanese general-purpose language model that can identify time-related common sense using the English dataset on temporal common sense, MC-TACO, translated into Japanese.
We have conducted experiments with several masking rates and settings: (i) masking random tokens, (ii) masking words of the answer part in the input (iii) masking temporal-related words, (iv) masking words with high attention score, and (v) masking words with high saliency score. Through our experiments, we have confirmed that high accuracy can be obtained in Japanese with the same settings as in English.