[2A6-GS-2] Machine learning

[2A6-GS-2-05] A Study on Performance and Efficiency Improvement of FT-Transformer

〇Tokimasa Isomura¹, Tomoki Amano¹, Ryotaro Shimizu¹, Masayuki Goto¹ (1. Waseda University)

Keywords:Transformer, Attention, Tabular Data, Explainable AI, DNN

In recent studies, Deep learning (DL) models demonstrate high performances on various datasets. FT-Transformer (FTT), which applies the Transformer model to tabular data, has been proposed as an effective DL model for tabular data. FTT performed higher on several datasets than gradient boosting models, the current mainstream for tabular data. Initially proposed for unstructured data, Transformer shows high performance by sensitively considering the relationships between all features (e.g., words and patch images) through the attention mechanism. However, the relationships between features in tabular data can be considered less complex than those in unstructured data such as documents and images. Therefore, we propose an improved FTT suitable for tabular data that does not excessively consider the unnecessary relationship between features in the Transformer's attention mechanism and improves performance and computational efficiency. We perform evaluation experiments on regression, binary classification, and multi-level classification tasks and show our model's effectiveness.