[4L3-OS-38] OS-38

[4L3-OS-38-04] User Evaluation of Datasets for Machine Learning Competitions

〇Takeaki Sakabe¹, Yuko Sakurai¹, Satoshi Oyama^2,3 (1. Nagoya Institute of Technology, 2. Nagoya City University, 3. RIKEN AIP)

Keywords:Machine Learning Dataset, Machine Learning Competition, User Evaluation

In this study, we evaluate user perceptions of datasets used in machine learning competitions. Datasets for such competitions are often selected in an unsystematic manner, making it challenging to choose datasets that align with participants' skill levels. To address this issue, we conducted a user study as part of the evaluation of a framework for automated dataset generation. In this study, participants trained and tested models using both existing datasets and those generated by the framework, and we collected test results based on algorithm selection and performance variations. The impact of datasets on participants' performance was examined using objective metrics. The results of our evaluation indicate that the generated datasets exhibit properties more suitable for educational competitions compared to existing datasets.