[4K3-IS-2f] Machine learning

[4K3-IS-2f-04] Learnability of Regular Languages in Language Models

〇Masaya Taniguchi¹, Naoki Negishi², Yusaku Nishimiya^4,1, Keisuke Sakaguchi², Kentaro Inui^3,2,1 (1. RIKEN, 2. Tohoku University, 3. Mohamed bin Zayed University of Artificial Intelligence, 4. University of Illinois Springfield)

Keywords:Formal Language, Learnability, Language Acquisition

This study explores the impact of the presentation order of positive and negative data on grammar acquisition in language models. We specifically focus on a text search problem, with the target grammar represented by a regular language. To conduct the study, we prepare two types of data: positive data, where sentences conforming to the target grammar are embedded within the text, and negative data, where such sentences are absent. Our findings demonstrate that both the sampling strategy for positive and negative data and the order in which these datasets are presented influence the language model's ability to acquire grammatical structures.