[3Xin2] Poster session 1

[3Xin2-12] Pretraining Memory Restricted RNN by Next Word Prediction for Fusing Symbols and Distributed Representations

〇Asami Ogawa¹, Yoshinobu Kano¹ (1.Shizuoka University)

Keywords:NLP, Symbols, RNN

We introduce the LSTM-Symbolic model, designed to handle symbolic representations within LSTM's memory cells for sequential processing, aiming to align model computations closer to human cognition and enhance explainability. This model utilizes the existing weights of the Embedding layer for converting between distributed and symbolic representations. We also introduced a concept of a memory ideal vector and defined a unique computation method for backpropagation. Moreover, we constructed a model hypothesizing a context-free grammar limited by memory through symbolization within a two-layer LSTM language model. Upon evaluating the generated syntax's validity through next-word prediction, we observed that models incorporating symbolic structures at the input layer significantly underperformed compared to others, despite identical layer counts. Remarkably, the proposed model matched the baseline's performance with fewer weight parameters, indicating its capability to grasp more syntax-essential structures, including promising local generation examples at sentence beginnings, suggesting an advancement in syntactic processing closer to human understanding and model transparency.