[3S4-GS-2] Machine learning:

[3S4-GS-2-03] Evaluation of Design Variable Interpolation in Evolutionary Model Merging

〇Rio Akizuki¹, Nozomu Yoshinari¹, Yuya Kudo¹, Yoichi Hirose¹, Toshiyuki Nishimoto¹, Kento Uchida¹, Shinichi Shirakawa¹ (1. Yokohama National University)

Keywords:Large Language Model, Model Merging, Evolutionary Model Merging, Evolutionary Computation

Model merging is a technique for combining deep learning models without additional training and enables us to integrate the abilities of multiple large language models (LLMs) into a single LLM. Evolutionary model merging optimizes merging parameters using evolutionary computation and can reduce manual trial and error. However, it is still computationally expensive due to the repeated evaluation of merged LLMs. In particular, it is difficult to ensure a sufficient number of evaluations when optimizing merging parameters for each layer. This study experimentally evaluates design variable interpolation in evolutionary model merging using Japanese and Chinese math tasks and a newly introduced surrogate benchmark. In design variable interpolation, only the merging parameters for several layers are used as design variables, and the merging parameters for other layers are computed using interpolation methods. The experimental results show that design variable interpolation could improve the merged model performance and accelerate the search process.