[M-IS02] Ocean plastics, an earth science perspective

[MIS02-P08] AI-Driven Beach Litter Detection with Digital Twin: Automatic Training Data Generation

*Sota Kuwada¹, Shinichiro Kako^1,2, Daisuke Sugiyama^2,1, Mitsuko Hidaka^1,2, Daisuke Matsuoka^2,1 (1.Graduate School of Science and Engineering, Kagoshima University, 2.Japan Agency for Marine-Earth Science and Technology)

Keywords:beach litter, deep learning, object detection, digital twin, drone, game engine

To implement effective measures against the issue of plastic marine debris, it is essential to establish methods for efficiently detecting and quantifying plastic marine debris. Hidaka et al. (2022) developed a model capable of detecting and quantifying coastal debris using semantic segmentation, a deep learning technique. However, this method required manually labeling each pixel in beach images with predefined class colors to create training data, which took approximately four months to generate 3,500 training images. To address this issue, we proposed a new approach that utilizes point cloud data obtained via UAVs and a game engine to automatically generate training data, and we evaluated its effectiveness and challenges (Kuwada et al., JpGU2024). As a result, the automatic generation of training data significantly reduced the time required for training data creation and demonstrated high detection performance for artificial debris. However, there remained a challenge: the detection performance for certain classes, such as natural debris and vegetation, was extremely low. To overcome this issue, the present study reexamined the training data generation method from the perspectives of “target classes for detection” and “placement of virtual models”. In our previous approach, similar to Hidaka et al. (2022), we classified objects into eight categories: artificial debris, natural debris, sandy beach, sea, sky, vegetation, structures, and background. While artificial debris was detected with reasonable accuracy, the detection performance for natural debris and vegetation was insufficient. Therefore, in this study, we attempted to improve detection performance by merging difficult-to-detect classes with other categories and optimizing the class combinations. Additionally, we reexamined the virtual beach constructed in the game engine. Our previous method involved adding surfaces to the point cloud data obtained by UAVs and placing them in the game engine as a virtual beach. However, the process of adding surfaces introduced discrepancies from the original point cloud data. In this study, we directly imported point cloud data and examined its impact on the model’s detection performance. Furthermore, for the virtual debris placed on the virtual beach, we refined the models based on the findings of Carmen et al. (2021) to create more realistic virtual debris and enhance detection performance. To compare the improved detection model with the model by Hidaka et al. (2022), we evaluated their performance using images on Rokudoji Beach in Toyama Prefecture, employing IoU, recall, and precision metrics. Additionally, to verify the generalizability of our approach, we applied it to Mitsu Beach in Shimane Prefecture, which has different environmental conditions from Rokudoji Beach. The results showed that our model demonstrated a certain level of detection performance for artificial debris not only at Rokudoji Beach but also at Mitsu Beach.