Evaluation and selection of CMIP6 GCMs for long-term hydrological projections based on spatial performance assessment metrics across South Korea

The Coupled Model Intercomparison Project 6 (CMIP6) was used to support the Sixth Assessment Report (AR6) of the Intergovernmental Panel on Climate Change (IPCC). The AR6 report has a much clearer picture of recent rapid, widespread, and intensifying changes in the climate. The effects of climate change begin to appear in various regions in multiple ways (Nunes & Dias 2022). An increase in drought severity and frequency (Wetter et al. 2014) and flash floods (Marchi et al. 2016) have been observed in Europe. Changes in climate extremes have been explored around the world over the last three decades, including Africa (Mason et al. 2010) and Asia. The increasing trend of heatwaves in East Asia has also been often discussed (Im et al. 2019). Kwon & Lall (2016) identified a century-scale drought during 2012–2015 in California, and strong fire activity has also been reported in Australia (Canadell et al. 2021).

Climate change is largely related to spatiotemporal variability in precipitation (Lindsey & Dahlman 2020). In particular, global warming increase of 1.5–2 °C or more would lead to an increase in extreme rainfalls (Masson-Delmotte et al. 2021). This emphasizes the need for a spatiotemporal precipitation modeling framework to guide the water management policies at different spatial scales (e.g., regional, local, or basin scale) under climate change scenarios (Akhter et al. 2017).

The Global Climate Models (GCMs) for simulating future climates were released under a series of CMIPs in different phases, and they are considered the primary source for interdisciplinary scientific climate research. CMIP3 and CMIP5 comprise 25 and 40 GCMs, respectively, and CMIP6, the most recent experimental design, has 55 GCMs (Eyring et al. 2016). The GCM simulations are principally used to analyze and simulate atmospheric circulation and climate on a global scale (Pour et al. 2018). Various studies indicated that CMIP6 simulations generally have better performance than CMIP5 in terms of bias in precipitation at the annual and seasonal scales (Gusain et al. 2020), indicating the CMIP6 GCMs better reproduce historical precipitation extremes observed at the regional and global scale than the CMIP5 GCMs. However, the inherent uncertainty in climate change scenarios from GCM simulations and the necessity of adopting a suitable subset of GCMs in the development of climate change scenarios have been addressed. In particular, the uncertainty of climate change scenarios was caused by the selection of GCMs, emission scenarios, downscaling model approach, and hydrological model structures, where a parameter estimation (Mandal et al. 2019) indicated that it is essential to evaluate the model performance of each GCM in terms of the ability to reproduce key statistics of historical precipitation. Cook et al.(2017) discussed the importance of choosing a suitable GCM for engineering applications, demonstrating differences in the impact assessment at different spatiotemporal scales. Therefore, the selection of a set of suitable GCMs from the CMIP6 inventory could be an essential reliable assessment using long-term hydrological simulation.

To evaluate and prioritize the best-performing GCMs, previous studies have used various machine learning models and statistical metrics based on information entropy, probability distribution function, a Bayesian approach, correlation analysis, clustering approach, hierarchical analysis of variance (ANOVA) models, symmetrical uncertainty approach, and recursive feature elimination model. More specifically, Wu et al. (2018) used standard deviation (σ), deterministic coefficient (DC), correlation coefficient (CC), relative error (RE), and root mean square error (RMSE) to evaluate six CMIP5 GCMs for precipitation in the Huaihe River Basin, China. He et al. (2019) assessed nine CMIP5 GCMs using CC and RMSE for temperature in China. In general, studies have combined different indicators to evaluate the model performance from different perspectives. Parth Sarthi et al. (2016) considered combined indicators, such as the Taylor diagram, skill score (SS), CC, and RMSE, while Raju & Kumar (2014) combined several indicators, such as average absolute relative error (AARE), normalized root mean square error (NRMSE), absolute normalized mean bias error (ANMBE), and SS. These performance indicators have been widely used because of their advantages for systematic comparison of different GCMs (Guilyardi et al. 2009), demonstrating effective assessment of model performance (Gu et al. 2015). However, there have been few attempts to investigate the spatial representation of a large-scale climate in selecting GCMs (Koch et al. 2018). More importantly, the spatial pattern of climate variables (e.g., precipitation and temperature) should be captured by GCMs for the region that is affected by monsoons (Srinivasa Raju et al. 2017). Spatial climate patterns provide important information with regard to the overall spatial coverage of areas affected by drought and flood in response to climate extremes (Khan et al. 2018; Salman et al. 2018).

The full use of available GCMs could effectively reduce the uncertainty associated with climate models, leading to a computationally inefficient process of extensive data (Lutz et al. 2016). Therefore, the selection of appropriate GCMs can provide an effective way for future long-term hydrological simulations. Several studies have been performed on the selection of subsets of GCMs (Kim et al. 2021) and their combinations (multimodel ensembles, MMEs) for selective GCMs (Suh et al. 2016) in South Korea. Recently, a selection process of representative GCMs for East Asia and South Asia has been adopted (Kim et al. 2020) and for the areas affected by the East Asian monsoons, including South Korea (Kim et al. 2021). They selected several pairs of CMIP5 and CMIP6 GCMs using the selection process, and Kim et al. (2020) explored 29 CMIP5 and 25 CMIP6 participating GCMs that have the required climate variables for hydrological simulations.

The Korean peninsula is affected by East Asian monsoons with a distinct spatiotemporal pattern of precipitation (Kwon et al. 2008; Uranchimeg et al. 2020), and the associated climate variability under climate change is likely to increase the risks of the hydrological system (Lima et al. 2021). Under these circumstances, a comprehensive assessment of future climate informed by CMIP6 GCMs is required for a formal hydrological risk analysis. This study aims to explore the model performance of CMIP6 GCMs over South Korea in terms of the capability to reproduce spatial climate patterns using Goodness-of-Fit (GoF) measures such as the SPAtial EFficiency metric (SPAEF) (Demirel et al. 2018), Kling–Gupta efficiency (KGE) (Gupta et al. 2009), Cramer's V (Cramér 1999), and fractions skill score (FSS) (Roberts & Lean 2008).

The paper is organized as follows. We introduce a brief overview of climate model selection for hydrological simulations and the main objectives of the study in this current section. In the following section, the research area and hydrometeorological datasets are summarized. The methodology is described in Section 3. Results and discussions are provided in Section 4. Finally, conclusions and suggestions for future research directions are presented in Section 5.

There is a clear difference in temperature during the four distinct seasons: Spring (MAM, March–April–May) with an average temperature of approximately 12 °C, Summer (June–July–August) with maximum temperatures exceeding 33 °C leading to heatwaves, Autumn (SON, September–October–November) with an average temperature of approximately 20 °C, and Winter (DJF, December–January–February) because of influences from the Siberian high-pressure system with an average temperature of approximately 1.2 °C. Based on the aim of the study, five major river basins of South Korea (Yeongsan, Seomjin, Geum, Nakdong, and Han) with 56 weather stations, as shown in Figure 1, were primarily considered. The names of the weather stations are summarized in Table 1.

The monthly precipitation data at 56 weather stations compiled by the Korea Meteorological Administration (KMA) were used to create spatial precipitation data using the inverse distance weighted (IDW) approach. The monthly simulated precipitation of 32 available CMIP6 GCMs for ensemble run r1i1p1f1 was obtained from http://cmip-pcmdi.llnl.gov/cmip6/ for the historical period of 1973–2014 in terms of considering the validity and reliability of GCM outputs. Based on the Coordinate Reference System for Korea, 32 GCMs in the CMIP6, as summarized in Table 2, were adopted in this research. For an effective comparison with point rainfalls, this study used a bilinear interpolation method to map the point rainfalls into a high spatial resolution 0.125° × 0.125° grid.

This study considered four indicators for the spatial representation of GCM models: Cramer's V, SPAEF, KGE, and FSS, to evaluate the association, variability, and error between observational data and GCMs output data.

The value of KGE varies between and 1, and a value closer to 1 indicates better agreement between observations and simulations.

The spatial pattern of the MAP from some of the GCMs (e.g., AWI-ESM-1-1-LR (0.4), E3SM-1-1-ECA (0.35), BCC-CSM2-MR (0.31), and CESM2-WACCM (0.28)), which is the model group with the highest correlation coefficients, appears to be similar to that of the observed, representing a better representation of the higher MAP in the lower Nakdong and Seomjin watersheds. Among all GCMs, AWI-ESM-1-1-LR showed the best performance in terms of representing the spatial distribution of MAP with a high CC of 0.4 that reproduces the precipitation in the Nakdong watershed. Alternatively, several GCMs appear to be a biased representation of the spatial distribution of MAP, highlighting the higher precipitation in the upper Nakdong watershed. Moreover, most GCMs fail to reproduce the spatial pattern of MAP in the eastern part of the Han River watershed, demonstrating underestimation or overestimation of the precipitation in the Han River and Geum River watersheds. The CESM2-WACCM tends to overestimate MAP in most Nakdong River watersheds with 1,400–1,700mm, while the observed MAP is about 900–1,100 mm. Furthermore, the model group with the lower correlation coefficients (e.g., GFDL-CM4, GISS-E2–1-G, EC-Earth3, and GFDL-ESM4) showed a spatially biased representation of the MAP. In summary, the GCMs have somewhat different simulation capabilities representing the spatial distribution of the MAP for each watershed. With this perspective, exploration of spatial performance over different CMIP6 GCMs can provide an effective means of selecting GCMs for hydrological study over the major river watersheds in South Korea.

In general, some GCMs may perform better than others for specific regions, even if they do not have higher spatial correlation coefficients. The GCMs with the higher correlation coefficients may indicate overall good performance, but they may not accurately reproduce the observed distribution of MAP (or MMP) with the entire range of precipitation seen in the individual region. Similarly, the GCMs with the lower correlation coefficients may not necessarily indicate poor performance, especially if the model can reproduce important regional features and climate variability. Therefore, exploration of the spatial performance and combining multiple metrics over a large number of CMIP6 GCMs provide an effective means of selecting GCMs for hydrological study over the major river watersheds in South Korea.

The ranking of GCMs based on each spatial metric is first demonstrated. A comprehensive rating metric (RM) integrating four metrics is then considered to find the overall ranking of 32 GCMs and the subset of GCMs for both MAP and MMP. The overall ranking of 32 GCMs with RM values for the MAP is summarized in Table 3. They can be divided into three groups based on their RM values: Tier-1 (0.61–0.73) with 9 GCMs, Tier-2 (0.42–0.58) with 14 GCMs, and Tier-3 (0.19–0.37) with 9 GCMs. The range of ranking for each GCM was found to be quite variable over four indicators, leading to a narrower range of the RM value due to the mixed effect of different rankings associated with each indicator. The different rankings over four indicators could be due to inconsistent simulation results in various aspects of the spatial pattern seen in the observed climate system. For example, the GCM models in Tier-1 (e.g., ACCESS-ESM1-5, FGOALS-g3, AWI-ESM-1-1-LR, and ACCESS-CM2) noticeably underestimate MAPs over different regions for each model. Moreover, the models in Tier-1 showed a limited capability in simulating the extreme rainfalls in the Han River, Seomjin River, and Nakdong River watersheds. As illustrated, approximately 60% of the annual precipitation in South Korea occurred during the monsoon season from June to the end of July. Thus, the GCM selection process based on model performance during the monsoon season would be more informative than that of the annual.

The RM values of 32 GCMs based on the ranking of various spatial metrics for the spatial MMP distribution performance are summarized in Table 4. According to the KGE metrics, the FIO-ESM-2-0 model appears to be most relevant with the highest value (0.47), and then CESM2-WACCM, CESM2, and CMCC-ESM2 models are pertinent with a relatively higher KGE value (>0.43). It can be concluded that those three models show better performance in terms of the KGE value with a higher association of the observed precipitation. Alternatively, AWI-ESM-1-1-LR, GISS-E2-1-G-CC, GISS- E2-1-H, and GISS-E2-2-H models have fairly lower KGE values (⁠0.10). Moreover, these GCMs appear to be relatively lower SPAEF values, demonstrating poor representation of the spatial pattern of the historical precipitation. Higher SPAEF values were observed for FGOALS-g3, FIO-ESM-2-0, and BCC-CSM2-MR, with a value greater than 0.4, indicating a better representation of spatial patterns of precipitation over South Korea. The best performance in terms of the FSS metric was observed for the FIO-ESM-2-0 model (0.66). Similarly, CMCC-ESM2, EC-Earth3-Veg, and CESM2-WACCM models have comparably greater FSS values than the others, with a value greater than 0.60. A relatively lower FSS value was found for AWI-ESM-1-1-LR, GISS-E2-1- G-CC, GISS-E2-1-H, and GISS-E2-2-H models. Alternatively, conflicting results were identified for GISS-E2-1-H, GISS-E2-1-G-CC, and AWI-ESM-1-1-LR models in terms of Cramer's V metric, with a favorable result. However, there is a very narrow range of Cramer's V between 0.38 and 0.48 in our case, indicating that the metric is not sensitive to the GCMs considered in this study. Moreover, Cramer's V metric is only based on the strength of the relationship between the binary variables (0 or 1) with the limited number of grids, such that the sensitivity of the metric could be less than others.

The overall ranking of 32 GCMs is also summarized in Table 4. The ranking of GCMs based on each metric is readily available, and the overall ranking combining four indices is also required to differentiate the subset of GCMs. In this perspective, the study employed the RM metric to effectively define the overall ranking of 32 GCMs. The four top-ranked GCMs based on their overall ranks in reproducing spatial distribution of precipitation over South Korea were identified. The range of the RM values can be classified into three categories: Tier-1 (0.72–0.84): FIO-ESM-2-0, CESM2-WACCM, CESM2, and CMCC-ESM2 models; Tier-3 (0.26–0.39): AWI-ESM-1-1-LR, GISS-E2-1-G-CC, GISS-E2-1-H, GISS-E2-2-H models; and Tier-2 (0.41–0.66): the remaining GCMs. In general, one may expect that rankings of the GCM model can vary over different metrics due to the limited representation of the regional climate system. In this perspective, the use of a comprehensive rating metric could offer a systematic way to better characterize model performance over several GCMs. In this study, four top-ranked GCMs (i.e., FIO-ESM-2-0, CESM2-WACCM, CESM2, and CMCC-ESM2) can be selected as the best-performing GCMs for long-term hydrological projections based on spatial performance assessment metrics across South Korea.

In general, it is clearly evident that there is a substantial difference in rainfall amount between the observed precipitation and that simulated by the best-performing GCMs in the eastern part of the Han River watershed and the nearly entire Nakdong River watershed. This bias may be due to the misrepresentation of the regional climate system. Thus, a relevant bias correction approach for minimizing systematic bias in GCM precipitations is needed prior to their usage for regional impact assessment of climate change on water resources.

This study does not perform bias correction for the data during the model selection process. However, one may expect that the bias correction approach could offer a better option in selecting a set of suitable GCMs. Furthermore, there will be a change in the ranking pattern under the bias correction process. From this perspective, this study included the bias-corrected results for best- and worst-performing GCMs based on a quantile mapping (QM) approach, which is the most common bias correction approach in the climate change study. The main idea of the QM-based bias correction for the simulated precipitation is to map its distribution to that of the observed. Thus, it should be noted that a marginal distribution of the bias-corrected precipitation is almost identical to the observed. As expected, the spatial distributions of bias-corrected MMPs for these GCMs are not discernable from those of the observed, regardless of the best- and worst-performing GCMs, as illustrated in Supplementary material, Figures S1 and S2. Under these circumstances, the bias correction for these GCMs is not recommended prior to model selection, and the ranking of GCMs under the bias correction process could be problematic.

The main objective of this study was to investigate the model performance of CMIP6 GCMs in South Korea in terms of capability to reproduce spatial climate patterns using GoF measures such as the SPAEF, KGE, Cramer's V, and FSS. Furthermore, this study proposed an integrated selection approach based on the four spatial performance assessment metrics to better demonstrate changes in precipitation in climate change studies. From this perspective, this study explored the high-performing GCMs among 32 available CMIP6 GCMs based on the GoF measures over the main watershed in South Korea for the historical period of 1973–2014.

It was found that a comprehensive rating metric (RM) approach was more effective to evaluate the overall ranking of the GCMs and then identify the desired GCMs due to the fact that the GCMs' performance differs from the spatial performance assessment metrics. Furthermore, the overall ranking of GCMs based on the KGE, FSS, and SPAEF metrics has relatively similar results, while inconsistent results based on the Cramer's V index are identified. In this perspective, the RM-based model selection process combining a variety of metrics could be more informative than a single metric by considering different aspects of model performance for hydrological study over the major river watersheds in South Korea. Finally, this study explored a suitable set of GCMs with RM values and further categorized the GCMs for both MAP and MMP. Among all GCMs, the co-selected GCMs based on RM values for representing the spatial distribution of both MAP and MMP are CESM2-WACCM and CMCC-ESM2.

According to the KGE, FSS, and SPAEF metrics, the FIO-ESM-2-0 model appears to be relevant, and the CESM2-WACCM, CESM2, and CMCC-ESM2 models have a better association with the observed precipitation. Furthermore, the best-performing GCMs showed more plausible spatial patterns than those observed. In contrast, the worst-performing GCMs showed more considerable differences in the spatial patterns observed in the precipitation during the historical period, and a significant underestimation of the precipitation was clearly marked over a large region. In this regard, this study explored the role of the bias correction approach in selecting GCMs and further demonstrated there will be a change in the ranking pattern under the bias correction process. Here, a QM-based bias correction approach was applied to the simulated MMPs from GCMs. The results showed that the spatial distributions of bias-corrected MMPs for these GCMs were almost identical to those of the observed, irrespective of the GCM performance. Thus, this study confirmed that bias correction is not necessarily required and recommended prior to model selection.

In summary, the selection of an appropriate subset of CMIP6 GCMs may be related to various factors, such as the variables of interest, the number of GCMs, spatial and temporal resolutions, and evaluation metrics. This study used a limited number of evaluation metrics tailored for the assessment of several GCMs, so that the selected GCMs may be changed with the use of other metrics. Therefore, future work should focus on the use of more expanded spatiotemporal evaluation metrics and the sensitivity of these metrics to the variation in model selection, providing incremental information over the existing model selection process.

This work was supported by Korea Environment Industry & Technology Institute (KEITI) through Water Management Program for Drought, funded by Korea Ministry of Environment (MOE) (2022003610003).

All relevant data are available from an online repository or repositories https://esgf-node.llnl.gov/search/cmip6/.

The authors declare there is no conflict.