[M-IS09] Ocean Plastics, an earth science perspective

[MIS09-10] Digital twin using a drone and a game engine for marine litter detection

*Sota Kuwada¹, Shin'ichiro Kako^1,2, Daisuke Sugiyama², Mitsuko Hidaka², Daisuke Matsuoka² (1. Graduate school of science and engineering, Kagoshima University, 2.Japan Agency for Marine-Earth Science and Technology)

Keywords:beach litter, deep learning, semantic segmentation, digital twin, drone, game engine

To conduct countermeasures against the marine litter problem effectively, it is necessary to establish an efficient method for observing and quantifying marine litter. Hidaka et al. (2022) constructed a deep learning model that can detect and quantify beach litter using semantic segmentation, one of the deep learning techniques. However, the training data used in their study was created by manually painting each pixel representing pre-specified classes on beach images, and it took approximately four months to create 3,500 sets of training data. Therefore, the purpose of the present study was to propose a completely novel method for automatically creating training data using a drone observation and a game engine, and to examine its usefulness and future challenges. The game engine developed by Epic Game, Unreal Engine 5 (hereafter UE5), was used to construct the virtual space, as UE5 has high-quality graphics and a physics engine, which are very useful for digital twins based on real space data. It is also capable of importing data in a variety of formats, including 3D point cloud data obtained from drone observations, enabling simulations that are in line with reality in virtual space. Furthermore, the system can automatically place and color objects, making it ideal for creating training data for semantic segmentation. In this study, a system was constructed to reproduce the beach terrain in UE5 from 3D point cloud data obtained by drone observation, automatically place virtual artificial and natural litter created in advance, and automatically output correct labels for each pixel. After that, as in Hidaka et al. (2022), by inputting these training data to the High-Resolution Network (HRNet; Sun et al., 2019; Wang et al., 2021), a deep learning model was constructed that can detect beach litter. This model and the model of Hidaka et al. (2022) were applied to beach litter images, and the IoU, recall and precision between them were compared to verify the usefulness and challenges of our system. The results show that the system is very effective in generating training data.