[AAS06-08] Direct Multi-class AUC Maximization for Forecasting Rapidly Intensifying Tropical Cyclones
Keywords:Intense tropical cyclones , intensity classification, AUC maximization
Intense tropical cyclones (TCs) have caused tremendous damage to human society. Most intense TCs experience rapid intensification (RI, Kaplan and DeMaria 2003). To predict TC intensification, the Statistical Hurricane Intensity Prediction System (SHIPS; DeMaria et al., 2005) has been developed and run in operational centers. SHIPS utilizes a multivariate regression model that predicts future intensity changes, i.e., the central pressure and maximum wind speed, for each time-window with predictors calculated from numerical models and observations.
Although SHIPS has been improved with lots of efforts (Shimada et al., 2018, DeMaria et al., 2014), it is essentially difficult to predict RI since the number of RI training samples is very small (~5%). More technically, in the training of regression models with imbalanced data, e.g., between rapid and moderate intensifications, the contribution of minor events, i.e., RI, would be vanished in the squared or absolute error due to the dominance of the majority events, i.e., moderate intensification.
To tackle such imbalanced problems in the TC forecast, we propose to formulate it as a classification problem where the time-window is classified into five future intensity levels, i.e., rapid weakening, moderate weakening, neutral, moderate intensification and RI. The direct maximization of Area Under Curve (AUC; Ueda and Fujino, 2018) is known to be a theoretically grounded approach to the imbalanced classification problem. Since AUC is the sum of true positive rates, to maximize AUC, the positive class, e.g., RI, has to be selected correctly. To apply the direct AUC maximization to the multiple intensity-level-classification, we originally introduce a multi-class variant of AUC, called “One-versus-the-Rest-AUC, ORAUC) where at each class, the class is treated as “positive” and the rest as “negative” to compute true-positive rates.
We show the effectiveness of our proposed method, ORAUC, through the forecast of five-intensity-level of TCs using the sequences of 48-hour-window-SHIPS-feature generated from TCs occurred between 1987 and 2017. We split the window-based data into the training (1987-2010), validation (2011-2013), and evaluation (2014-2017) data. The label of each window is annotated based on the thresholds of the change of central pressure in 48-hour; thresholds are statistically determined as 5-, 25-, 75- and 95-percentiles of data (see the top-left of Fig.).
We compare three methods: SHIPS (linear regression), neural network with cross-entropy loss (CE), and neural network with ORAUC (proposed method). The figure depicts the distribution of central pressure changes, the overview of the entire flow of our proposed method, and the experimental evaluation. As for the evaluation metric, we use precision and recall for each class and mean F1 score over classes. As the table (see the bottom-right of Fig.) shows, our proposed method improves the recall (true positive rates) for the class of rapid weakening and intensification where there are only 5% of data since the AUC of each class is maximized. Above all, our proposed method obtains the best mean F1 score over classes, indicating that our proposed method, ORAUC, could improve the performance of RI prediction by handling the imbalanced data problem well.
Although SHIPS has been improved with lots of efforts (Shimada et al., 2018, DeMaria et al., 2014), it is essentially difficult to predict RI since the number of RI training samples is very small (~5%). More technically, in the training of regression models with imbalanced data, e.g., between rapid and moderate intensifications, the contribution of minor events, i.e., RI, would be vanished in the squared or absolute error due to the dominance of the majority events, i.e., moderate intensification.
To tackle such imbalanced problems in the TC forecast, we propose to formulate it as a classification problem where the time-window is classified into five future intensity levels, i.e., rapid weakening, moderate weakening, neutral, moderate intensification and RI. The direct maximization of Area Under Curve (AUC; Ueda and Fujino, 2018) is known to be a theoretically grounded approach to the imbalanced classification problem. Since AUC is the sum of true positive rates, to maximize AUC, the positive class, e.g., RI, has to be selected correctly. To apply the direct AUC maximization to the multiple intensity-level-classification, we originally introduce a multi-class variant of AUC, called “One-versus-the-Rest-AUC, ORAUC) where at each class, the class is treated as “positive” and the rest as “negative” to compute true-positive rates.
We show the effectiveness of our proposed method, ORAUC, through the forecast of five-intensity-level of TCs using the sequences of 48-hour-window-SHIPS-feature generated from TCs occurred between 1987 and 2017. We split the window-based data into the training (1987-2010), validation (2011-2013), and evaluation (2014-2017) data. The label of each window is annotated based on the thresholds of the change of central pressure in 48-hour; thresholds are statistically determined as 5-, 25-, 75- and 95-percentiles of data (see the top-left of Fig.).
We compare three methods: SHIPS (linear regression), neural network with cross-entropy loss (CE), and neural network with ORAUC (proposed method). The figure depicts the distribution of central pressure changes, the overview of the entire flow of our proposed method, and the experimental evaluation. As for the evaluation metric, we use precision and recall for each class and mean F1 score over classes. As the table (see the bottom-right of Fig.) shows, our proposed method improves the recall (true positive rates) for the class of rapid weakening and intensification where there are only 5% of data since the AUC of each class is maximized. Above all, our proposed method obtains the best mean F1 score over classes, indicating that our proposed method, ORAUC, could improve the performance of RI prediction by handling the imbalanced data problem well.