[2Win5] Poster session 2

[2Win5-05] Exploring Decoder-Based Tabular Transformer Models with Piecewise Linear Embeddings

〇Taisei Tosaki^1,2, Nanae Aratake¹, Yuji Okamoto¹, Eiichiro Uchino¹, Ryosuke Kojima^1,3, Yasushi Okuno^1,2 (1.Kyoto University, 2.RIKEN Center for Computational Science, 3.RIKEN Biosystems Dynamics Research)

Keywords:Tabular Transformer, Piecewise Linear Embedding, Generative model

Deep learning, and in particular Transformer, has been successful in the fields of computer vision and natural language processing, where unstructured data is predominant. In recent years, there has been a transformation of structured tabular data into unstructured strings, which has been applied against Transformer, which is used in large language models. In this case, tabular data consists of a sequence of label names and their value pairs, which are a mixture of text and numerical values. However, the computational expense of these methods on large scales hinders their practical application. This study proposes a novel Decoder-Based Tabular Transformer, utilising sentence embedding and piecewise linear embedding of numerical values, to address this challenge. The efficacy of this approach is validated through its application to tabular data comprising both sentences and numerical values. The proposed model demonstrated a correct response rate of 0.856 on the US annual income prediction benchmark set of the UC Irvine Repository, which is comparable to the performance of the existing method (0.876). Future work should compare the proposed model with previous studies that utilised methods to convert tabular data to strings.