JSAI2024

Presentation information

Poster Session

Poster session » Poster session

[4Xin2] Poster session 2

Fri. May 31, 2024 12:00 PM - 1:40 PM Room X (Event hall 1)

[4Xin2-112] Development of a dynamic resource allocation system for ML pipelines

〇Kazuhiro Saito1, Kei Yonekawa1, Shigeki Muramatsu1, Mori Kurokawa1 (1.KDDI Research, Inc.)

Keywords:ML pipeline, MLOps, Dynamic resource allocation

In a society where various machine learning models are used in practical applications, the importance of managing the machine learning pipelines (ML pipelines), which is a series of processes from data processing to training and inference, is increasing. A ML pipeline platform provides an environment to manage ML pipelines and these computer resources, such as CPU, memory, and GPU. However, current platforms require pre-configuration of computer resources to be used by ML pipelines, and cannot take into account the actual resource usage, resulting in the allocation of extra resources. In this paper, we propose a system that dynamically predicts the amount of computer resources used in each phase of the ML pipeline and executes the ML pipeline with the minimum necessary resources, and evaluate the efficiency of resource use for transfer learning.

Authentication for paper PDF access

A password is required to view paper PDFs. If you are a registered participant, please log on the site from Participant Log In.
You could view the PDF with entering the PDF viewing password bellow.

Password