[M-AG33] Satellite Land Physical Processes Monitoring at Medium/High/Very High Resolution

[MAG33-P02] A global Swin-Unet Sentinel-2 surface reflectance-based cloud and cloud shadow detection algorithm for the NASA Harmonized Landsat Sentinel-2 (HLS) dataset

*Haiyan Huang¹, David P Roy¹, Hugo De Lemos¹, Yuean Qiu¹, Hankui K Zhang² (1.Center for Global Change and Earth Observations, Michigan State University, East Lansing, MI 48824, USA, 2.Geospatial Sciences Center of Excellence, Department of Geography and Geospatial Sciences, South Dakota State University, Brookings, SD 57007, USA)

Keywords:cloud, cloud shadow, deep learning, Sentinel-2, HLS, Swin-Unet

The most recent Version 2.0 NASA Harmonized Landsat Sentinel-2 (HLS) data provides global coverage atmospherically corrected surface reflectance with a 30 m cloud and cloud shadow mask derived using the Fmask algorithm applied to top-of-atmosphere (TOA) reflectance. In this study we demonstrate, as have other researchers, low Sentinel-2 Fmask performance, and present a solution that applies a deep learning Swin-Unet model to the HLS surface reflectance to provide unambiguously improved cloud and cloud shadow detection. The model was trained and assessed using 30 m HLS surface reflectance for the 13 Sentinel-2 bands and corresponding CloudSEN12+ 10 m annotations, that define cloud, thin cloud, clear, and cloud shadow classes, and is the largest publicly available expert annotation set. A total of 8,672 globally distributed 5 x 5 km CloudSEN12 annotation data sets were used, 7,362 to train the model, 464 for internal model validation, and 846 to independently assess the classification accuracy. The HLS Sentinel-2 Fmask had F1-scores of 0.832 (cloud), 0.546 (cloud shadow), and 0.873 (clear), and the Swin-Unet model had higher performance with F1-scores of 0.891 (cloud and thin cloud combined), 0.710 (cloud shadow), and 0.923 (clear) despite the use of surface and not TOA reflectance. The Swin-Unet thin cloud class had low accuracy (0.604 F1-score) likely due to atmospheric correction issues and thin cloud variability that are discussed. The comprehensively trained model provides a solution for users who wish to improve the HLS Sentinel-2 cloud and cloud shadow masking using the available HLS Sentinel-2 surface reflectance data pending HLS reprocessing.