An integrated method for extended-range prediction of heavy Precipitation process in the flood season over Hunan Province based on S2S models

The sub-seasonal to seasonal (S2S) prediction research project is jointly established by the World Weather Research Program (WWRP) and World Climate Research Program (WCRP) under the World Meteorological Organization (WMO). The sub-seasonal prediction, which has multiple time scales, is the key interface between the weather forecast and seasonal prediction. Applying the numerical weather forecast to short-term climate prediction can bridge the gap between medium-range weather forecasts and seasonal predictions (WMO 2013).

In recent years, benefiting from the continuous developments of models, assimilation technology, and ensemble methods, the prediction skills of the S2S models have been remarkably improved. For example, the forecast lengths are 44 and 46 days for the National Center for Environment Prediction (NCEP) and the European Centre for Medium-Range Weather Forecasts (ECMWF), respectively, indicating a higher prediction skill of the latter (CMA 2018). China Meteorological Administration (CMA) actively participates in the international cooperation of the S2S prediction project, launches China's sub-seasonal timescale forecast research (Qi & Rong 2014), successively develops several versions of climate system models, and gradually realizes their operational application (Wu et al. 2010, 2013, 2014; Wu 2012; Hu et al. 2020), which is helpful to improve the capability of global forecast service (Hu et al. 2019). At present, the CMA model has a forecast length of up to 60 days.

Scholars have comprehensively evaluated the prediction skills for precipitation and temperature (Tian et al. 2017), cold surges (Li et al. 2017), climate change process in summer of East Asia (Liang & Lin 2017), and summer monsoon anomaly in Australia (Marshall & Hendon 2015) on the sub-seasonal timescale. In addition, Bombardi et al. (2017) assessed the sub-seasonal predictability of three global S2S models on the start and end dates of rainy seasons in South America, East Asia, and northern Australia. Zhao et al. (2016) comprehensively evaluated the performance of the BCC_CSM1.2 S2S model. These studies have revealed that the S2S models have certain predictability, but they are still difficult to perform skilled predictions (Lin et al. 2019).

At the same time, some studies (Rashid et al. 2011; Kim et al. 2014; Neena et al. 2014; Vitart 2014) have shown that the qualitative and quantitative prediction skills of the S2S models for precipitation in the extended range vary with the forecast month. For example, qualitative prediction skills are higher in the flood season, especially in midsummer, whereas quantitative prediction skills are the lowest in midsummer. From the perspective of qualitative prediction skills on a sub-seasonal timescale, models of the CMA and BOM (Bureau of Meteorology Australia) perform the best in predicting extreme precipitation in May and June to September, respectively (Pang et al. 2021).

Located in the Subtropical Monsoon Climate Zone of Central China, Hunan Province has experienced frequent rainfall in the flood season, which is easy to cause natural disasters such as mountain torrents, mudslides, and urban waterlogging. In addition, there are one to three concentration periods for rainfall in Hunan Province during the flood season every year. During these periods, the short interval of precipitation processes and large rainfall intensity will cause basin floods easily. Sub-seasonal to seasonal prediction can extend the forecast length of the heavy rainfall process, which can help the government departments to switch from passive response to active mitigation of disasters. It is also crucial to improve the meteorological service capacity and build the first line of defence in disaster prevention and mitigation. Since the S2S prediction is performed on the global scale, there is still a certain degree of deviation in the local-scale prediction, such as in Hunan, especially for the insufficiency of prediction stability by the single model. Thus, based on the products of S2S models from the CMA, ECMWF, and NCEP, the extended range prediction skills of the heavy precipitation process in the flood season over Hunan Province were evaluated. On this basis, an adjusted method for single-model prediction and an integration method for multi-model predictions were established in this study, which has important scientific and social application values.

The remainder of this article is organized as follows. Section 2 introduces the observation and prediction data of precipitation, the definition and scoring rules of the heavy precipitation process, and the adjusted and integration methods for the S2S models. Section 3 presents the assessments of original models, adjusted and integration methods. Finally, the conclusions and discussion are presented in Section 4.

The S2S models used in this study are from the CMA, ECMWF, and NCEP. Table 1 lists the model details, including forecast length, resolution, number of ensemble members, and forecast frequency. The CMA S2S model has a forecast length of 60 days and a horizontal resolution of about 0.5° × 0.5°, with four ensemble members and a forecast frequency of one time per day. The ECMWF S2S model has a forecast length of 46 days and a horizontal resolution of 16 km within 0–15 days and 32 km after 15 days, which has 51 ensemble members and makes forecasts twice in 1 week. The NCEPS2S model has a forecast length of 44 days and a horizontal resolution of about 1° × 1°, with 16 ensemble members and the same forecast frequency as the CMA model. The prediction skills of MJO for the aforementioned three models are 24, 23–27, and 20 days, respectively (Peng et al. 2021).

The gridded daily precipitation from model predictions was interpolated to 96 national stations in Hunan Province by the bilinear method. Then the predicted precipitation was matched with the observation from April to August during 2017–2020 to calculate the prediction score of the heavy precipitation process to evaluate the prediction skills of the three models.

The standard of moderate rain in Hunan is that the precipitation exceeds 10 mm in a day, which will have a certain impact on traffic, agriculture, and so on. Therefore, the daily precipitation at a single station above 10 mm is considered heavy precipitation in this study. The heavy precipitation process is determined by the standard that there are at least six stations in the whole province having heavy precipitation in a single day. The heavy precipitation process can be classified as a short process and a long process. The short process refers to the process with a duration of 2 days or less, and the long process lasts for 3 days or more.

At present, the meteorological department in China has not yet established a national standard for the evaluation of extended range prediction of the heavy precipitation process. Due to the actual needs of the local vocational work, we developed the scoring rules in this article based on actual service needs and practical experiences in Hunan. The following are specific rules.

For short precipitation processes, the statistical period is from the day before the first day of the process to the day after the last day. If heavy precipitation is observed at six stations or more on any single day in the statistical period, it is considered that the process is correctly predicted; otherwise, it is a false alarm prediction. For the prediction process that lasts only one day, the score is 100 for correct prediction, and the score is 0 for false alarm prediction.

When the precipitation process is predicted to be 2 days, if the observed heavy precipitation process is completed within the model forecast period, the prediction score is recorded as 100. If only part of the observed process occurs in the model forecast period, the prediction is scored as 70. If no heavy precipitation process is observed during the model forecast period, the score is marked as 0, indicating a false alarm prediction.

For a long precipitation process, the statistical period is the same as the forecast period. In this period, if the number of observed heavy precipitation days is greater than or equal to 1/3 of the forecast period, the prediction is considered correct, and the score is recorded as 100; otherwise, it is a false alarm prediction with the score being 0.

The days outside the statistical periods are regarded as the days without predicted heavy precipitation. These days, if the heavy precipitation process is observed, it is regarded as a missing prediction.

It should be specially noted that if there are multiple prediction samples in different forecast leading times for a certain forecast future period, the average scores of all samples during the corresponding forecast future period were taken as the values of ⁠, ⁠, and ⁠. The original prediction score of a single model (or original score for short) was calculated by Equations (1) and (2).

Since there are some systematic errors in the model prediction, the heavy precipitation prediction was adjusted by changing the threshold value of identifying heavy precipitation. The specific procedures are as follows: (1) take the threshold values from 6 to 15 mm with an interval of 1 mm to determine the heavy precipitation day; (2) calculate the prediction score under each threshold value; (3) select the threshold value with the highest prediction score as the optimal threshold to determine the heavy precipitation process (hereinafter referred to as optimal threshold method). Finally, the adjusted prediction scores of each model (or adjusted scores for short) at different forecast leading times for certain forecast future periods are calculated by Equations (1) and (2).

It should be noted that analysis has been conducted on the range of threshold. If the threshold is less than 5 mm, it will increase the false alarm rate. On the contrary, if it exceeds 16 mm, it will increase the missing rate.

⁠, ⁠, and ⁠; ⁠, ⁠, and ⁠; and ⁠, ⁠, and represent the weight coefficients of the CMA, NCEP, and ECMWF S2S models at different forecast leading times(1–5, 6–10, and 11–15 days), respectively, which were determined by the model's prediction skill for heavy precipitation process.

When the of a certain day is equal to or larger than 0.5, the day is determined as an integrated heavy precipitation day; otherwise, there is no heavy precipitation on that day. Then, the prediction scores of multi-model integration (or integrated scores for short) at different forecast future times are calculated by Equation (1).

As described in Section 2.2.1, the false alarm prediction refers to the situation that there is a heavy precipitation process in prediction when no heavy precipitation process is found in observation. The missing prediction refers to the situation that there is a heavy precipitation process in observation but no heavy precipitation in prediction. Therefore, the false alarm rate is the ratio of the number of false alarm predictions to the total number of predicted heavy precipitation processes, and the missing prediction rate refers to the number of missing predictions divided by the total number of observed processes.

For the average score of each model from April to August, the score of the CMA model is 69.2, which ranks the first among the three, followed by 66.9 for the ECMWF model and 64.9 for the NCEP model.

To eliminate the systematic errors of the CMA, ECMWF, and NCEP models, different thresholds were selected to calculate the prediction scores of heavy precipitation process during April to August for each model (shown in Tables 2–4). It can be found that the optimal thresholds from April to August are 7, 7, 6, 14, and 6 mm for the CMA model (shown in Table 2), while they are 8, 7, 6, 6, and 6 mm for the ECMWF model (shown in Table 3) and 6 mm, 6 mm, 9, 6 and 6 mm for the NCEP model (shown in Table 4), respectively.

In summary, the predicted precipitation is systematically lower than the observation, resulting in optimal thresholds being mostly lower than the original threshold of the observed heavy precipitation process (10 mm). With the increase of the threshold, the score shows a general downward trend.

Based on the optimal thresholds, the predictions of the heavy precipitation process of CMA, ECMWF, and NCEP models were adjusted. Table 5 shows that the adjusted scores of the CMA and NCEP models are higher than the original scores in both forecasts of 11–20 and 21–30 days for the future. For the whole extended range of 11–30 days, both the average adjusted scores of the CMA and NCEP models are larger than the original scores, for 5.9 and 5.0, respectively. However, the average adjusted score of the ECMWF model slightly decreases by about 0.5 in the whole extended range. This is because the prediction skills of ECMWF in the next 1–10 days are also considered when determining the optimal threshold. After the threshold is determined, the prediction skills of ECMWF in the next 1–10 days are improved, but decreased in the next 11–30 days. In general, the skills in the next 1–30 days are still improved.

According to the optimal threshold method, the scores of each model in predicting the heavy precipitation process in the next 1–30 days for different forecast leading times were analysed. It is found that the prediction score increases with the decreasing of forecast leading time. The prediction at the forecast leading times of 1–5 days has the highest average score of 70.2, followed by 69.5 at 6–10 days and 67.8 at 11–15 days. For different models, the CMA model has the highest prediction score, with an average of 71.6, followed by the NCEP model with an average score of 68.4, and the ECMWF model has the lowest average score of 67.5 (shown in Table 6).

Considering the prediction skills of different models, the weight coefficients of models during the process of multi-model integration were determined by the prediction scores. Specifically, higher prediction scores and smaller forecast leading times correspond to larger weight coefficients. Therefore, the coefficients of the CMA, NCEP, and ECMWF models and the forecast leading times of 1–5, 6–10, and 11–15 days were both assigned with 1, 0.8, and 0.6, respectively. The weight coefficient for a certain model at certain forecasting leading time was obtained by multiplying the two corresponding coefficients (shown in Table 7).

Specifically, the original scores of the CMA, ECMWF, and NCEP models in the range of next 11–20 days are 67.8, 66.1, and 67.4, respectively, with an average of 67.1, while the adjusted scores are 71.1, 65.0, and 69.2, respectively, with an average of 68.4, slightly higher than the average of original scores. The integrated score is 77.4, which is 10.3 and 9.0 higher than the average of original and adjusted scores, respectively.

For the next 21–30 days, the original scores of the CMA, ECMWF, and NCEP models for heavy precipitation process prediction are 64.0, 65.2, and 58.2, respectively, with an average of 62.5. The adjusted scores are 72.5, 65.2, and 66.5, respectively, with an average of 68.1, higher than the average of original scores (62.5) by 5.6. The integrated score is 72.8, which is 10.3 and 4.7 higher than the averages of original and adjusted scores over the three models, respectively.

The original scores of the CMA, ECMWF, and NCEP models in the whole extended range are 65.9, 65.6, and 62.8, respectively, with an average of 64.8. The adjusted scores are 71.8, 65.1, and 67.8, respectively, with an average of 68.2, higher than the average of original scores (64.8) by 3.4. The integrated score is 75.1, which is 10.3 and 6.9 higher than the averages of original and adjusted scores over the three models, respectively.

False alarms and missing rates are also important indicators to evaluate the prediction skills of models.

It was found that the false alarm rates of the original CMA, ECMWF, and NCEP models are 8.0, 16.9, and 17.1%, respectively, with an average of 14.0%, while the false alarm rates of the adjusted method decrease to 6.2, 8.7, and 10.0%, with an average of 8.3%, which is 5.7% smaller than the original average rate. The false alarm rate of the integrated method is 5.4%, which decreases by 8.6 and 2.9% with respect to the original and adjusted average rates, respectively.

The missing rates of the original CMA, ECMWF, and NCEP models are 37.2, 38.7, and 39.9%, respectively, with an average of 38.6%. The missing rates of the adjusted method are 34.8, 37.9, and 39.0%, respectively, with an average of 37.2%, which is 1.4% smaller than the original average rate. The missing rate of the integration method is 33.9%, which is 4.7 and 3.3% smaller than the original and adjusted average rates, respectively (shown in Table 8).

In this study, we first defined the heavy precipitation process and the scoring rules for heavy precipitation prediction according to the characteristics of precipitation in Hunan Province. Based on the forecast precipitation data during 2017–2020 from three operational models (CMA, ECMWF, and NCEP) that participated in the S2S prediction project and the observational daily precipitation data from 96 national meteorological stations in Hunan, the monthly prediction skills from April to August of the models were evaluated. According to the prediction scores under different thresholds of heavy precipitation, the optimal threshold of each model for each month was obtained to calculate the adjusted scores of different models at different forecast leading times. The weight coefficients of the models were further obtained based on the adjusted scores and were used to determine the multi-model integration method and calculate the integrated scores. Finally, by defining the false alarm prediction and missing prediction, the false alarm and missing rates of original models, and adjusted and integration methods were analysed. The main conclusions are as follows.

The original prediction scores of the heavy precipitation process for each model are relatively high during May to June and low in April, July, and August, with the highest score of 80.5 in June and the lowest score of 56.6 in July. The CMA model has the highest average score, followed by the ECMWF and NCEP models.

The adjusted scores of the CMA and NCEP models are higher than the original scores in all forecasts of 11–20 days, 21–30 days, and the whole extended range for the future, while the adjusted scores of the ECMWF model are lower. For the whole extended range of 11–30 days, the adjusted scores of both the CMA and NCEP models increase by about 5.9 and 5.0, but it slightly decreases by about 0.5 for the ECMWF model.

The integrated scores in the forecasts of 11–20 and 21–30 days for the future are 77.4 and 72.8, respectively, both higher than the original and adjusted scores of single-model prediction. The integrated score in the whole extended range is 75.1, which is 10.3 and 6.9 higher than the averages of original and adjusted scores over the three models, respectively.

The false alarm rate of the integration method is 5.4%, which is 8.6 and 2.9% smaller than the average rates of the original models and adjusted methods, respectively. Similarly, the missing rate of the integration method is 33.9%, which is 4.7 and 3.3% smaller than the average missing rates of original models and adjusted methods, respectively.

The primary purpose of this study is to establish a relatively stable method for forecasting heavy precipitation processes as far ahead as possible to provide services for the government in disaster prevention and mitigation. However, it should be pointed out that the prediction of heavy rainfall in the extended range is a very complex work. Taking this study as an example, there is still a lot to be improved as discussed in the following.

The scoring rules of the heavy precipitation process only consider the duration of the heavy precipitation process, but ignore the spatial distribution. This is because current models have low accuracy in predicting the location of heavy precipitation processes. However, it is relatively accurate to predict the time period when heavy precipitation processes occur on a larger scale. Therefore, government decision-makers are more concerned about whether there has been heavy rain or the frequency of heavy rain in the administrative region, and release corresponding warning information (such as control of reservoir levels). Anyway, this will bring great limitations and make the prediction unable to fully meet the requirement.

In determining the weight coefficients of models, although the prediction skills at different forecast leading times are considered, where smaller forecast leading times and higher prediction scores correspond to larger weight coefficients, there are still many subjective factors. It is necessary to adopt different methods to obtain the optimal scheme to calculate the weight coefficients in the future.

We thank Nanjing Hurricane Translation for reviewing the English language quality of this paper.

C. M. analysed the data. J. P. put forwarded the idea. C. M. and J. P. wrote the manuscript. All authors revised the manuscript.

This study was supported by the Support Plan of Enterprise Innovation and Entrepreneurship Team for science and technology in Hunan and the Capacity Building Project of Hunan Meteorological Bureau in 2020 (Grant No. NLJS04).

All relevant data are included in the paper or its Supplementary Information.

The authors declare there is no conflict.