Development of category-based scoring support vector regression (CBS-SVR) for drought prediction

Drought prediction provides early warnings of drought development and valuable drought mitigation information for stakeholders. The data-driven methods widely used for drought prediction include time-series analysis (Rao & Padmanabhan 1984; Mishra & Desai 2005; Modarres 2007; Fernández et al. 2009; Durdu 2010), linear regression (Liu & Juárez 2001; Panu & Sharma 2002; Barros & Bowden 2008; Sun & Kim 2016) and nonlinear regression (Hwang & Carbone 2009; Liu & Hwang 2015), artificial neural network (ANN) (Mishra & Desai 2006; Mishra et al. 2007; Morid et al. 2007; Barua et al. 2012; Santos et al. 2014; Yang et al. 2015), Markov chain analysis (Lohani & Loganathan 1997; Cancelliere et al. 2007; Paulo & Pereira 2007, 2008; Sharma & Panu 2012), and probabilistic forecasting (Madadgar & Moradkhani 2013; Hao & Singh 2016).

Regression models are used if the predictand (drought index) is continuous. For drought prediction, support vector machine (SVM) for regression or support vector regression (SVR) has been suggested since it has some advantages over other prediction models. For example, the SVR can overcome some limitations of ANN such as local maxima and overfitting, and outperform ANN to some extent for drought prediction (Ganguli & Reddy 2014). It also has the ability to learn from a much smaller dataset for training and is capable of handling a large number of variables (Hao et al. 2018). The success or failure of a prediction model depends on various factors such as the selection of predictors. Shamshirband et al. (2020) showed a relatively worse performance of SVR than other drought prediction models when meteorological drought indices were used to predict hydrological drought. Mohamadi et al. (2020) compared different drought prediction models and found that a hybrid SVM showed a relatively lower capability in predicting droughts among the models tested in their study.

There are various hyperparameters in an SVR model. Penalty factor determines the trade-off between the model complexity and the training error (Joachims 2002). Parameter epsilon controls the width of the epsilon-insensitive zone, which is used to fit the training data. The optimal value of epsilon scales linearly with gamma (Schölkopf & Smola 2002), a Kernel parameter used to reduce the model space and control the complexity of the solution (Kisi & Cimen 2011). Therefore, tuning these hyperparameters is crucial in defining decision boundaries and the success of an SVR model. These hyperparameters have been selected by trial and error in some studies for drought prediction (Belayneh et al. 2014; Feng et al. 2019). Cross-validation methods have been also used for tuning these hyperparameters. Grid search (Larochelle et al. 2007) is one of the well-accepted cross-validation methods in SVR (Bergstra & Bengio 2012) for drought prediction (Deo et al. 2016a, 2016b; Deo et al. 2017).

To assess the prediction skill of a regression model, the regression results are commonly compared with a reference (i.e., actual values). Mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE), and coefficient of determination (R²) are among the most commonly-used performance measures (Sundararajan et al. 2021). Although these performance metrics enhance the preciseness and predictive skills of models, they may cause misclassification (hereafter miscategorization). For instance, the objective in a cross-validation effort for tuning hyperparameters in SVR is to select the values of hyperparameters so that lower MAE, MSE, and RMSE or higher R² are achieved. Although it can reduce the difference between the actual and predicted values of drought indices and increase the mathematical preciseness of the prediction model, it may also lead to miscategorization of the predicted drought. This provides misinformation for stakeholders and causes waste of budgets and efforts, while they take inappropriate actions for preparedness and mitigation for the predicted drought category.

Drought categorization converts a large volume of data (drought indices) into a category that represents a measure of severity and facilitates ‘apple to apple’ comparisons over time, according to the National Center for Environmental Information (NCEI 2020). Compared with drought indices values, interpretation of drought categories is simple and can easily be understood by stakeholders. Thus, stakeholders such as decision makers mostly care about drought categories rather than the values of drought indices. Any miscategorization in drought studies can mislead the stakeholders. Thus, defining a threshold level for drought categorization is a crucial process (Bazrkar et al. 2020; Hao et al. 2016a).

One can avoid drought miscategorization by using the classification framework such as support vector classification (SVC) and logistic regression. A logistic regression model works for a binary drought category (drought or wet). Since most of the existing drought information systems are based on multiple drought categories, the logistic regression model needs to be modified to be employed for drought prediction (Regonda et al. 2006; Hao et al. 2016b, 2016c). However, the potential limitation is the large number of parameters for the prediction of multiple drought categories (Hao et al. 2018).

Accurate predictions of drought categories are crucial for stakeholders to decide on how to be prepared for potential upcoming droughts. However, using the existing standard performance metrics such as RMSE for training SVR-based drought prediction models potentially leads to miscategorization of droughts since such metrics minimize the difference between the actual and predicted values by accounting for the numerical differences only without considering the categorical discrepancies. Depending on whether the predicted value is lower or greater than the actual value, the predicted value can be closer to the actual value in two directions. The ideal condition is to remove or minimize the numerical differences in a direction so that the predicted value is located in the same category as the actual value. However, this ideal condition is less achievable or can be achieved through certain improvements in the concept behind the prediction models (e.g., the overfitting issue in ANN can be resolved in SVR by applying structural risk). This study focuses on improving the capability of a specific drought prediction model by avoiding miscategorization and aims to develop a new category-based scoring approach for tuning hyperparameters and to improve the prediction model training by accounting for the differences in drought categories, rather than the values of drought indices. Specifically, the objective of this study is to use the concept of classification in cross-validation of an SVR model to avoid miscategorization in drought prediction. A novel category-based scoring SVR (CBS-SVR) method is developed and compared with the traditional SVR and SVC since these models share some conceptual similarities. SVR is a numerical model with a potential of miscategorization. Thus, a comparison of SVR and CBS-SVR can shed light on the improvement achieved by the category-based scoring employed in this study. SVC has a classification nature and its comparison with CBS-SVR helps assess their performances.

SVR was developed by Vapnik (1995) to reduce the generalization error by using the structural risk minimization. A regression function for a set of sampled points from input and target vectors is estimated by tuning hyperparameters. There exist different Kernel functions such as linear, polynomial, sigmoid, and nonlinear radial basis function (RBF) that can be used to increase the dimensional space in SVR. Depending on the nature of the input data, different Kernel functions can be employed, and thus different hyperparameters need to be tuned. In the current study, various methods were tested, and the nonlinear RBF was selected. Thus, the hyperparameters included gamma (γ), cost (C), and epsilon (ε). Parameter C determines the trade-off between the model complexity and the training error. Parameter ε controls the width of the ε-insensitive zone and can affect the number of support vectors used to construct the regression function. Parameter γ reduces the model space and controls the complexity of the solution (Kisi & Cimen 2011). These three hyperparameters in SVR (i.e., γ, C, and ε) need to be tuned. Cross-validation has been widely used to identify the values of these hyperparameters. In this method, a combination of hyperparameters is selected based on a specific type of scoring. In this study, a novel category-based scoring method was developed and used in CBS-SVR. Grid search cross-validation was also used for tuning these hyperparameters. A wider range was tested for each hyperparameter and then narrowed down based on the results. The narrower range was set as the initial range of the hyperparameter in grid search cross-validation. C values varied among 1, 5, 10, 15, 20, and 25; ε values included 1, 0.1, 0.01, 0.001, and 0.0001; and γ values were 0.1, 0.5, 1, 1.5, 2, and 5. The best combination was eventually selected based on the new category-based R scoring.

Training, validation, and testing are three phases in the development of an SVR-based prediction model. Training is conducted on the training dataset. If the evaluation of performance on the validation dataset is successfully proceeded, the final evaluation can be done on the testing dataset. There are two major problems if the dataset is divided into three parts: (1) a decrease in the number of samples by wasting too much data and (2) dependence of the results on the random choice for the training and validation datasets. k-fold cross-validation solves these problems by dividing the dataset into training and testing sets. The training set is further divided into k-folds and the validation process is repeated k times so that k−1-folds are used for training and 1-fold is used for the primary evaluation. The averaged value of performance measures in the k loops is the performance measure of k-fold cross-validation. The minimum value of k is 2, by which the dataset is divided into two subsets for training and test. No maximum value is determined for k. The higher the k value, the more training data and the more complex the problem. The optimum k value (equal to 5) was selected in this study, as shown in Figure 1. Among a number of combinations of hyperparameters that go through the cross-validation, the value with the lowest error is selected and set for the testing period of the model (Figure 1).

The selected combination of hyperparameters is used in SVR to define the weights and intercept. Figure 2 shows two cases of miscategorization by using RMSE and how CBS-SVR can avoid this issue. D4, D3, D2, D1, and D0 are exceptional, extreme, severe, moderate, and abnormal droughts, respectively; N indicates a normal condition; and W0, W1, W2, W3, and W4 are representatives of abnormal, moderate, severe, extreme, and exceptional wet conditions, respectively. The thresholds are selected based on the traditional drought categorization. In case 1, the value predicted by SVR using the RMSE scoring is closer to the actual value but it is in a different drought category. The value predicted by CBS-SVR has a higher RMSE but it is in the same category as that of the actual value. In case 2, as a result of overfitting in other cases (e.g., case 1), the difference between the predicted value by SVR and the actual value is larger than that of the value predicted by CBS-SVR. Thus, the RMSE scoring causes miscategorization, while CBS-SVR successfully avoids this problem.

Drought categorization plays a crucial role in this study since it determines the hyperparameters for SVR and thus impacts drought prediction. Traditional categorization is based on fixed threshold levels, which need to be varied by time and location (Mishra & Singh 2010; Bazrkar et al. 2020). Bazrkar et al. (2020) proposed a customized drought categorization (CDC) and employed a cell-by-cell-based analysis to incorporate both spatial and temporal distributions of drought. A joint probability distribution and conditional expectation were used to estimate the average probability of occurrence of each drought category. K-means (KM) clustering was further used in the CDC to derive variable thresholds. Two normal distributions (one for dry and another for wet) were observed in their study. A Gaussian mixture (GM) model is suggested when two normal distributions are combined. Therefore, GM can be employed for drought categorization in the CDC. Traditional drought categorization (McKee et al. 1993) is based on the normal distribution (N). Svoboda et al. (2002) employed the percentile method to define the categories thresholds. The same categories were used in this study, including exceptional, extreme, severe, moderate, and abnormal dry conditions as well as the normal condition. In the traditional drought categorization, drought indices values of −3.00, −2.00, −1.00, and −0.50 are, respectively, the thresholds for extreme, severe, moderate, and abnormal drought categories. 0.50, 1.00, 2.00, and 3.00 are used for abnormal, moderate, severe, and extreme wet categories, respectively. The values between −0.50 and 0.50 are categorized as a normal category. The threshold values for exceptional dry and wet categories cannot be defined unless a long study period is chosen. Since the study period in this research was limited, the threshold values for exceptional dry and wet categories were predefined as −3.50 and 3.5, respectively. These values were chosen to be in accordance with the values for extreme dry and wet categories. The variable threshold levels for drought categories in the CDC were developed to vary by time and geographic locations and were determined by KM or GM for these 11 categories. In this study, the performance of KM was compared with the traditional categorization (McKee et al. 1993) and GM.

To assess the performance of CBS-SVR, it was first compared with the traditional SVR with the RMSE-based scoring. Then, CBS-SVR was compared with SVC. To measure and compare the performances of the models, a confusion matrix was extended from binary-class to multiclass and then different performance metrics were used.

RMSE has been widely used as scoring in cross-validation for tuning hyperparameters in SVR and for the evaluation of the performances of other drought prediction models. However, there is a risk of miscategorization if hyperparameters are selected based on RMSE. The traditional RMSE-based SVR was modified to overcome the issue of miscategorization by developing CBS-SVR in this study. Due to its classification nature, SVC is potentially capable of avoiding miscategorization. To quantize the predictive skills of each model, the performance of CBS-SVR was compared with that of SVR and SVC.

The main difference between SVR and SVC is in their loss functions. The loss function in SVC just assures that the prediction is greater than 1 if the predicted value is positive and less than −1 if the predicted value is negative (Figure 3). However, instead of minimizing the observed training error, the loss function in the SVR tries to minimize the generalization error bound so that a generalized performance is achieved.

The confusion matrix (Stehman 1997) and accuracy measures such as the area under the curve of a receiving operating characteristic (AUC-ROC) were originally developed for two-category or binary models. Various approaches have been used to extend the original binary models to multiclass models. One versus all (also referred to as one versus rest) and one versus one (Provost & Domingos 2000) are among the well-known approaches that use binary classification algorithms for multiclass classification. One versus all generates fewer binary classifiers and thus has relatively lower accuracy, but it is less complex. Therefore, to avoid complexity, one versus all was used in this study to evaluate the performances of the multi-category drought prediction models. Figure 4 shows how a confusion matrix for two categories (dry/wet) is extended to the one for 11 drought categories. The AUC-ROC and micro, macro, and weighted average (Fawcett 2001, 2006) of F1 scores were used to obtain overall performances of CBS-SVR, SVR, and SVC.

Precipitation, temperature, actual evapotranspiration, and a deficit variable (DFCT) were selected as the predictors. The DFCT was defined as the difference between precipitation and actual evapotranspiration. The target variables consisted of three types of drought indices at different time scales (monthly, seasonally, and semiannually) including (1) multivariate hydroclimatic aggregate drought index (HADI) (Bazrkar et al. 2020) and snow-based hydroclimatic aggregate drought index (SHADI) (Bazrkar & Chu 2022), (2) bivariate standardized drought indices based on the difference between precipitation and actual evapotranspiration (SPEI) and the summation of snowmelt and rainfall, and (3) univariate standardized drought indices based on precipitation, surface runoff, baseflow (BF), soil moisture, snowpack, and snow water equivalent (SWE). In an application to the Red River of the North Basin (RRB), the North American Land Data Assimilation System (NLDAS) (Xia et al. 2012) data in 1979–2010 and 2011–2016 were, respectively, used for the training and testing of the CBS-SVR model.

In this study, a CBS-SVR model was developed. Sundararajan et al. (2021) reviewed several studies on applications of machine learning for drought prediction and suggested using multiple drought indices for prediction, instead of a single index. Following this suggestion, 10 drought indices were used to evaluate the performance of CBS-SVR. The multivariate HADI and SHADI (Bazrkar et al. 2020; Bazrkar & Chu 2022) distinguished drought and snow drought in cold climate regions. The R-mode correlation-based principal component analysis (PCA) was employed to aggregate rainfall (RF), snowmelt (SM), surface runoff (R), and soil water storage (SWS) in the calculation of HADI. Precipitation and snowpack (SP) were used, instead of rainfall and snowmelt in HADI, to calculate SHADI to enhance the capability of identifying snow drought (Bazrkar & Chu 2022). Two bivariate standardized drought indices were also used. The standardized difference between precipitation and potential evapotranspiration was used to develop SPEI by Vicente-Serrano et al. (2010). Homdee et al. (2016) and Joetzjer et al. (2013) argued that the use of actual evapotranspiration showed higher consistency with hydrological drought indices and it considered land use and vegetation cover. In this study, actual evapotranspiration was used to calculate SPEI. The standardized snowmelt and rainfall index (SMRI) (Staudinger et al. 2014) was also estimated. The univariate standardized drought indices used in this study included standardized precipitation index (SPI) (McKee et al. 1993), standardized runoff index (SRI) (Shukla & Wood 2008), standardized soil moisture index (SSMI) (Xu et al. 2018), standardized snowpack index (SSPI), standardized snow water equivalent index (SSWEI) (Huning & Aghakouchak 2018), and standardized baseflow index (SBFI) (Bazrkar & Chu 2020). These drought indices cover a wide range of hydroclimatic variables and different types of drought indices, and hence using all of them can reassure the capability of the CBS-SVR model.

Depending on the specific drought index used as the target variable in the prediction model, different types of predictor(s) were used. Table 1 shows the predictands and their associated predictors. Precipitation (P) was used as a predictor for the prediction of all drought indices. Temperature (T) was added for the prediction of SPEI, SMRI, SSPI, and SSWEI, where besides precipitation, temperature also played a key role in the identification of drought. Actual evapotranspiration (ET) and the DFCT were also added in the prediction of HADI, SHADI, SSMI, SRI, and SBFI.

The U.S. part of the RRB (Figure 5) is a typical cold climate region in the Northern Great Plain, which covers over 90,000 km² in the states of Minnesota (MN), North Dakota (ND), and South Dakota (SD). According to the NCEI data (2021), the cold season of 2020 was the driest and the 10th warmest cold season, and March 2021 was the driest and the 4th warmest March in the last 127 years in most of the climate divisions (CDs) in ND. This sets an alarm for more severe droughts in the RRB. According to Palmer Z-index for March 2021 (Figure 5), CDs 3202, 3203, and 3205 were the driest, and CD 2101 was the wettest in the RRB. However, according to the 5-year mean of Z-index, CD 2101 was also among the driest CDs in the RRB, besides CDs 3202, 3203, and 3205.

A great portion of the RRB is covered by cultivated croplands. Therefore, drought has significant socioeconomic impacts on the population in the RRB. One of the devastative droughts in the RRB occurred in the 1990s. That drought was referred to as a snow drought since the RRB experienced the lowest precipitation and the highest temperature in the cold seasons in that dry decade. However, the 1930s drought known as the Great Drought was due to low precipitation in the warm seasons (Bazrkar & Chu 2022). Therefore, different drought indices with the capabilities of addressing different types of drought are required. In this study, the performances of different drought indices were assessed in the CBS-SVR model in the RRB.

To select the best categorization method, the traditional categorization based on the normal distribution and the CDC with KM and GM clustering methods based on the 10 drought indices were compared. The R scoring was not used in this step for the selection of the best categorization method. Instead, the RMSE scoring, a well-accepted metric, was used for cross-validation of these models. Table 2 shows the best, worst, and average of AUC-ROCs and the weighted, micro, and macro F1 scores for these models. A categorization method was selected as the best if the majority of performance measures showed the highest values for that specific categorization method. KM was selected as the best for SHADI and SMRI. For SHADI, micro AUC-ROC, weighted F1 score, and micro F1 score were 0.71, 0.38, and 0.47 for KM clustering, respectively. All metrics, except macro F1 score, were better than those of the other categorization methods. Therefore, KM was selected as the best categorization method for SHADI. For SMRI, all metrics indicated that KM had better performances than the traditional drought categorization and CDC with the GM clustering method. Micro AUC-ROC, weighted F1 score, micro F1 score, and macro F1 score were 0.89, 0.80, 0.80, and 0.43, respectively. Traditional categorization was selected as the best for SPEI, SPI, SRI, SSMI, and SBFI. For HADI, AUC-ROC and the micro F1 score suggested traditional categorization, while the weighted and macro F1 scores showed KM as the best categorization method. Unlike other indices, the best results for SSPI were obtained when GM clustering was applied. According to the results, a slight to significant improvement was achieved by changing the employed categorization method. For instance, a significant increase (0.18) in the weighted F1 score was observed when GM was switched to KM in the categorization of drought based on SRI. The most significant improvement (0.21) was obtained for the same measure and index when the traditional normal categorization was used instead of GM. There was also an improvement in the performance of the model (an increase of 0.08 in the micro F1 score) when the traditional normal drought categorization based on SSPI was changed to GM. It was concluded that depending on the selected drought index or indices, different performances and results were obtained when different drought categorization methods were employed.

Table 3 shows the variable thresholds for the drought categories based on the drought indices. Different variable thresholds were obtained for each drought index. The variable thresholds for HADI and SHADI were close. This result was in accordance with the finding by Bazrkar & Chu (2022), where the output from a grid-based hydrologic model (GHM) (Chu et al. 2019) in a different study period (2003–2007) was used. The clustering method was used to derive these variable thresholds based on the probabilities of occurrences of those values considering both spatial and temporal distributions of droughts. For example, the lower- and upper-class limits for the normal category in the traditional categorization were −0.5 and 0.5, respectively. However, the lower class limits for the normal category were as low as −1.05 for SHADI and −1.03 for SSPI, and the upper-class limit was as low as −0.33 for SSPI when GM was used for clustering. Since the range of each drought index was different, these different variable thresholds reassured that each drought category was defined based on the temporal and spatial frequencies of the drought index values.

The best categorization method for each drought index was selected and then SVR based on RMSE and CBS-SVR based on R scoring were compared for the drought indices. The results of this comparison are shown in Table 4. Unlike the results for SRI, SSMI, SSPI, and SSWEI, better results were obtained when R was used, instead of the traditional RMSE for cross-validation of SVR based on HADI, SHADI, SPEI, SMRI, SPI, and SBFI. Shamshirband et al. (2020) also found that SVR had a relatively lower capability in drought prediction than other models tested in their study.

Although the overall results (areal average) for all grids showed that R scoring was better than RMSE for the drought prediction models based on most drought indices (6 out of 10), four cases occurred for different grids: (1) R scoring performed better than RMSE; (2) RMSE performed better than R scoring; (3) there was no difference in the performances of R scoring and RMSE based on AUC-ROC; and (4) there was no difference in the performances of R scoring and RMSE based on the standard performance measures (i.e., R², MAE, MSE, and RMSE). Figure 6 shows the AUC-ROC for the 11 categories of drought in the SVR model based on SRI for grid 466, where R outperformed RMSE with the highest degree among all grids (an example for case 1). The micro-average of AUC-ROC increased from 0.66 to 0.73 when R scoring was used for cross-validation of SVR for SRI. Table 6 also shows an increase in R² (from 0.63 for R scoring to 0.70 for RMSE) and decreases in MAE, MSE, and RMSE for grid 466. When RMSE was used (Figure 10), it caused overfitting in some months and underfitting in other months (e.g., March 2011, September 2011, and September 2015). The value of the actual SRI in February 2015 was −0.47, indicating a normal condition based on the best categorization method for SRI (i.e., traditional normal). The predicted value by the R scoring for the same month was 0.41 and still in a normal condition. However, the predicted value by using RMSE was 0.54 and in an abnormal wet condition. This was an example of miscategorization caused by using RMSE. However, the CBS-SVR model avoided this kind of miscategorization by using R scoring.

The micro- and macro-averages of a specific metric are slightly different in their calculations. In the macro-average computation, the metric is computed independently for each class and then the average is taken by treating all classes equally. However, in the micro-average computation, the contributions of all classes are aggregated to compute the average metric. Therefore, the values of ‘nan’ for macro-average in Figures 6–9 were attributed to the severer dry/wet categories that did not occur in the study area and study period, and thus their AUC-ROC were indicted as ‘nan.’ If lower values are assigned for the threshold variables of exceptional dry and wet categories, the probabilities of occurrence of these exceptional categories are higher and fewer ‘nan’ values can be observed. Ignoring these values of nan, the AUC-ROC values for moderate wet (0.58–0.71), abnormal wet (0.53–0.70), normal (0.64–0.76), and abnormal dry (0.54–0.61) increased when R scoring was used. For moderate dry, a decrease in AUC-ROC from 0.72 to 0.64 and for severe drought a slight decrease from 0.50 to 0.49 were observed. The same value for AUC-ROC (0.5) for extreme drought was observed for both RMSE and R scoring. The highest type I and type II errors occurred for normal and abnormal wet categories, respectively (Table 5). However, the magnitudes of errors in R scoring were lower than those of RMSE. For these categories, both precision and recall increased when R scoring was used. For abnormal wet, the precision and recall values increased from 0.87 to 0.92 and from 0.76 to 0.79, respectively. For the normal category, higher increases in precision (from 0.70 to 0.79) and recall (from 0.77 to 0.86) were observed.

Figure 7 shows the AUC-ROC for grid 399, where RMSE outperformed R with the highest degree among all grids (an example for case 2). The micro-average of AUC-ROC for RMSE decreased from 0.78 to 0.69 when R scoring was used. The exceptional dry and wet and extreme wet categories were not observed, and thus their associated values were depicted as nan in Figure 7. The AUC-ROC values for severe wet (0.49) and extreme dry (0.5) categories remained unchanged. Except for the abnormal dry and moderate dry categories, decreases in the AUC-ROC values were observed for all other drought categories. The highest type I and type II errors occurred for abnormal wet and normal categories, respectively (Table 4). However, the magnitudes of errors in R scoring were higher than those of RMSE. For these categories, both precision and recall decreased when R scoring was used. For abnormal wet, precision and recall decreased from 0.88 to 0.82 and from 0.88 to 0.83, respectively. For the normal category, higher decreases in precision (from 0.85 to 0.73) and recall (from 0.75 to 0.61) were observed. According to Figure 10, the major differences were related to April 2011, 2012, and 2013, as well as March 2014, when the predictions based on RMSE were closer to the actual values. Table 6 also shows that R² increased and MAE, MSE, and RMSE decreased after the application of RMSE.

Figure 8 shows the AUC-ROC for grid 400, where the same AUC-ROC was obtained when the performances of RMSE and R were compared (an example for case 3). However, except for the normal category, the AUC-ROC varied for different drought categories. Increases in the AUC-ROC values from 0.52 to 0.58, from 0.48 to 0.69, and from 0.49 to 0.50 were observed for abnormal, moderate, and severe drought categories, respectively. In contrast, the AUC-ROC values decreased for moderate (from 0.78 to 0.68) and abnormal wet (from 0.66 to 0.62) categories. Table 5 shows that the highest type II error occurred with the same value (13 out of 72 months) in the normal category. This means that 13 out of 72 months were categorized as a normal condition but the model predicted a different category. The precision (0.83) and recall (0.69) had the same value in this category. When RMSE was used, the highest type I error occurred in the abnormal wet/dry and moderate dry categories with a value of 7 out of 72. However, when R scoring was used, the highest type I error occurred in the abnormal and moderate wet categories with a value of 8. Based on Figure 10, the major difference was related to March 2014, when R scoring performed better than RMSE. Table 5 also shows that based on R², MAE, MSE, and RMSE, SVR was better than CBS-SVR.

Figure 9 shows the AUC-ROC for grid 401, where R², MAE, MSE, and RMSE were equal when the performances of RMSE and R scoring were compared (an example for case 4). Although the R², MAE, MSE, and RMSE values were the same, there was a slight difference between AUC-ROCs. AUC-ROC increased from 0.72 to 0.73 when R scoring was used. This difference was attributed to the slight increases in AUC-ROC of the normal and abnormal drought categories, while AUC-ROC was exactly the same for all other drought categories. The highest type I and II errors also occurred in the same categories (i.e., moderate dry and normal, respectively). Table 6 indicates that the values of R², MAE, MSE, and RMSE were exactly the same. Figure 10 also shows the similar values of SRI in grid 401.

Table 7 shows the results of comparison between SVC and CBS-SVR for the drought indices. According to the results, CBS-SVR outperformed SVC based on HADI, SHADI, SPEI, SMRI, SPI, SSMI, SSWEI, and SBFI. Two exceptions were observed for SRI and SSPI. Overall, CBS-SVR had better performances since a category-based scoring was used for tuning the hyperparameters for cross-validation. Although all the metrics indicated better performances of CBS-SVR over SVC when HADI, SHADI, SPEI, SMRI, SPI, SSMI, SSWEI, and SBFI were used, the differences between the weighted F1 scores were remarkable. In the multiclass classification, a weighted average is preferred if there is a class imbalance (i.e., one class has more occurrences than others). The weighted F1 scores for CBS-SVR were 0.457, 0.415, 0.651, 0.804, 0.975, 0.339, 0.543, and 0.434 for HADI, SHADI, SPEI, SMRI, SPI, SSMI, SSWEI, and SBFI, respectively. Lower weighted F1 scores were observed for SVC for all drought indices except for SRI and SSPI. The weighted F1 scores for SVC for HADI, SHADI, SPEI, SMRI, SPI, SSMI, SSWEI, and SBFI were 0.433, 0.394, 0.640, 0.757, 0.960, 0.267, 0.527, and 0.377, respectively. The superiority of CBS-SVR over SVC is related to their different loss functions. Unlike SVC, which minimizes the observed training error, the loss function in CBS-SVR tries to minimize the generalization error bound so that a generalized performance is achieved. Thus, although SVC can avoid drought miscategorization, CBS-SVR predicts drought more accurately while avoiding drought miscategorization.

Regardless of the drought categorization methods used in the SVR model, the best performance was observed when SPI was used in the drought prediction model (Table 2). An AUC-ROC of 0.99 was achieved when SPI was used for all three types of drought categorization methods. The weighted and micro F1 scores for all cases were equal to 0.97, indicating high accuracy and precision of the SPI-based model. In contrast, when SSMI was used in the prediction model, the performance of the model was relatively poor, indicating the worst performance among the drought indices (Table 2). The values of micro AUC-ROC were 0.66, 0.64, and 0.60 for the traditional drought categorization, CDC with KM, and CDC with GM, respectively.

Similarly, for both CBS-SVR and SVC, SPI and SSMI had the best and worst performances, respectively. For SPI and SSMI, the values of micro AUC-ROC for CBS-SVR with traditional drought categorization were 0.986 and 0.656, respectively (Table 4). The values of micro AUC-ROC of SVC for SPI and SSMI were 0.980 and 0.653, respectively (Table 7). Thus, regardless of the type of the drought prediction models, the employed drought index is influential on the results, which justifies the application of multiple drought indices, as suggested by Sundararajan et al. (2021), for testing and comparing the performances of different prediction models and eventually assessing the performance of CBS-SVR developed in this study.

The poor model performance for SSMI can be attributed to the complexity of the hydrological processes associated with soil moisture and its relationships with other hydroclimatic variables that were not included in the employed predictors. Inclusion of more predictors other than precipitation, evapotranspiration, and temperature or consideration of other large-scale climate variables such as sea surface temperature (e.g., Lima & AghaKouchak 2017) and ocean oscillation indices (e.g., Deo et al. 2017) can improve the modeling results. Moreover, the RRB is a cold climate region and the frozen ground considerably impacts soil moisture. However, the frozen ground process is not fully considered in the NLDAS model. The better performance of the SPI-based prediction model can be attributed to a less complex relationship between precipitation (predictor) and SPI. The methodology proposed in this research is a feasible solution for the models with poor performances, and the results demonstrate how a category-based scoring method used for tuning hyperparameters can effectively improve drought prediction.

Using the standard performance metrics such as RMSE for scoring in cross-validation of non-categorical prediction models can result in overfitting and miscategorization of droughts. In this study, a category-based scoring method (CBS-SVR) was proposed. Due to the crucial role of the thresholds for defining drought categories, KM and GM clustering methods were tested and compared with the traditional categorization method based on AUC-ROC and F1 score. The traditional drought categorization and CDC with KM clustering showed better results. The traditional categorization was selected as the best method for most of the standardized drought indices including SPEI, SPI, SRI, SSMI, and SBFI. In addition to the drought categorization method, the employed drought indices played critical roles in the determination of the thresholds for drought categories and hence their accurate predictions.

The CBS-SVR with R scoring outperformed the SVR with RMSE scoring in most cases, specifically if the target variables were multivariate and bivariate drought indices. A grid-based analysis also confirmed that R was a better scoring method since it avoided overfitting and miscategorization. CBS-SVR also performed better than SVC for eight out of ten drought indices employed in this study. The results suggested that using the proposed category-based scoring method for tuning hyperparameters in cross-validation of SVR successfully avoided overfitting and drought miscategorization.

There are some limitations that can be addressed in the future research. Since this study focused on comparing the performances of a new category-based scoring method and the traditional standard approaches in a numerical prediction model (i.e., SVR), specific ranges of the hyperparameters were selected for cross-validation. Further analyses can be performed for a wider range of each hyperparameter and other numerical prediction models. In this study, KM and GM were adopted to derive variable threshold levels for drought categorization. Performances of other clustering methods can be tested in the future studies. Other clustering methods can potentially affect the results and lead to different variable thresholds, and thus the performances of drought prediction models can also be explored. In this study, local climate variables such as precipitation, temperature, actual evapotranspiration, and a deficit variable were adopted as the predictors. The accessibility and availability of the data were the primary reason for choosing these variables. The other large-scale climate indices that reflect the atmosphere-ocean circulation pattern (e.g., sea surface temperature, southern oscillation index, and North Atlantic oscillation) can be tested in the future studies.

This material is based upon work supported by the National Science Foundation under Grant No. NSF EPSCoR Award IIA-1355466. The North Dakota Water Resources Research Institute also provided partial financial support in the form of a graduate fellowship for the first author.

The authors declare that they have no conflict of interest.

All relevant data are included in the paper or its Supplementary Information. Some or all data, models, or code generated or used during the study are available in a repository online in accordance with funder data retention policies. The datasets used for this research are available at the National Center for Atmospheric Research Staff (Eds) (2019), The Climate Data Guide: NLDAS: North American Land Data Assimilation System: https://climatedataguide.ucar.edu/climate-data/nldas-north-american-land-data-assimilation-system. The modeling data generated from this study will be eventually uploaded to the UND Scholarly Commons and will be available for any interested readers.