[3E1-GS-10] AI application:

[3E1-GS-10-03] Workers’ behavior analysis based on unsupervised segmentation of multimodal time series data

〇Kuya Kondo¹, Issei Saito¹, Masatoshi Nagano¹, Satoshi Kamiya², Wataru Fujita², Toshiyuki Hatta^1,2, Tomoaki Nakamura¹ (1. The University of Electro-Communications, 2. Advanced Technology R&D Center, Mitubishi Electric Corporation)

Keywords:Behavior analysis, Unsupervised segmentation, Hidden semi-Markov model, Gaussian process

In analyzing human work behaviors at production sites, traditional methods primarily rely on visual observation, which is time-consuming and labor-intensive. To address this issue, we proposed a method that uses a Gaussian Process Hidden Semi-Markov Model (GP-HSMM) to segment workers’ continuous motions into discrete work behavior classes in an unsupervised manner. However, since this method relies solely on motion data, it has a limitation where motions with similar characteristics, even if they represent different behaviors, tend to be segmented into the same class. To overcome this issue, this paper proposes a novel method that incorporates multimodal information—including object data such as tools and components used by workers—in addition to motion data, for unsupervised segmentation. This approach enables more accurate segmentation, distinguishing behaviors involving similar motions based on the objects used. Preliminary evaluations were conducted using a simple work behavior dataset. The results demonstrated that the proposed method, utilizing multimodal information, achieved higher segmentation accuracy compared to the conventional method that relied only on motion data.