Evaluation of the BMA probabilistic inflow forecasts using TIGGE numeric precipitation predictions based on artificial neural network

Flood is widely regarded as the most frequent and severest natural disasters worldwide, which brings countless deaths each year (ICOLD 2006; Kussul et al. 2008; Uddin et al. 2013; Gao et al. 2015; Kwon & Kang 2016; Chen & Singh 2017). To mitigate the negative impacts of flood disasters, numerous structural measures (e.g. dams, levees, retention basins and detention basins) and non-structural measures (e.g. flood forecasting and alarms, reservoir operation and flood insurance) are utilized in practice, among which flood forecasting takes a vital role since it can offer flood information beforehand and hence set aside time for flood operation decision making and evacuation activity (Guo et al. 2004).

Flood forecasting is an important tool for reducing vulnerabilities and flood risk and forms an important ingredient of the strategy to ‘live with floods’, thereby contributing to national sustainable development (WMO 2010). Traditional flood forecasting is usually implemented by simply inputting precipitation predictions and other necessary elements to drive the hydrologic model. Thus the forecasting results are deterministic and can hardly describe the uncertainty of flood events. To overcome the shortages of deterministic flood forecasting, the probabilistic flood forecasting based on a probability distribution function to describe the future discharge has been proposed and gained great focus by many authors in recent years (Van Steenbergen et al. 2015; Bellier et al. 2016; Hardy et al. 2016; Liu et al. 2016; Li et al. 2017; Todini 2017). As a conclusion of previous studies, probabilistic flood forecasting outperforms deterministic flood forecasting in several aspects: (1) probabilistic flood forecasting can quantify uncertainty and enable the decision makers to hedge against the probability of forecast results; (2) probabilistic flood forecasting usually has a longer lead time and can provide more timely flood information; and (3) probabilistic flood forecasting usually has higher skills than deterministic forecasting.

The basic requirements of flood forecasting usually consist of specific precipitation predictions and calibrated hydrologic models. With flood protection and awareness continually rising on the political agenda, a strong demand has generated for achieving high quality precipitation prediction to obtain flood forecasts with sufficient lead time (Bao et al. 2011). Since the early 1900s, numerical weather prediction (NWP) techniques have been developed and offered a new approach to generate precipitation prediction (Bjerknes 1904). Cloke & Pappenberger (2009) reviewed the importance of NWP in flood forecasting and emphasized that NWP must be used for achieving reliable 2–15 days ahead flood forecasting. The Interactive Grand Global Ensemble (TIGGE) is part of the Hydrologic Ensemble Prediction Experiment (HEPEX) and ‘a project designed to develop, demonstrate and evaluate a multi-model, multi-analysis and multi-national ensemble prediction system’, intending to provide reliable, skillful, open source hydrologic forecast procedures by using emerging weather and climate ensemble forecast techniques (WMO 2005). By coupling the TIGGE precipitation predictions with hydrologic models, the lead times can be extended from a few hours to several days, thus holding great benefits for flood protection and decision making. Huang et al. (2010) conducted uncertainty assessment by probabilistic flood forecasts with the TIGGE data and applied the cost-benefit analysis based on the forecasting results. Zhao et al. (2011) compared the TIGGE data from different meteorological centers and found that the ECMWF performs better than the CMA and NCEP in the upper Huaihe basin. Bao et al. (2011) used the TIGGE data to drive the Xiananjiang model and conducted probabilistic forecasts at Xixian station in the Huaihe basin. The results showed that by coupling the TIGGE data with hydrologic models, flood events could be efficiently identified several days ahead. Liu & Xie (2014) used Bayesian model averaging (BMA) to the TIGGE precipitation predictions in the Huaihe basin and successfully applied the probabilistic precipitation to heavy-rain warning. Fan et al. (2015) used ensemble flood forecasts of the TIGGE data to verify hydropower reservoir inflows and showed that ensemble flood forecasts are more consistent than deterministic ones in terms of sequential decisions. Coustau et al. (2015) assessed the impacts of the ECMWF NWP products on streamflow forecasts and revealed that the atmospheric forcing is especially significant for streamflow forecasts of small catchments. These studies give evidence that the TIGGE data are valuable for flood forecasting and disaster relief decision making.

Among the many types of research on the TIGGE database, few have coupled it with a hydrologic model and evaluated its ability for generating probabilistic inflow forecasts in large catchments. Thus the objective of this paper is to establish a probabilistic inflow forecasting scheme in Danjiangkou basin based on the TIGGE data and the BMA method, and evaluate the skills of the probabilistic inflow forecasts. Three TIGGE NWP models, i.e. CMA, NCEP and ECMWF, were selected for the case study. The raw NWP data were downloaded and post-processed automatically from the official website of the ECMWF (http://apps.ecmwf.int/datasets/data/tigge). The remainder of this paper is organized as follows. The next section gives a brief introduction of Danjiangkou basin and presents the data used in this study. The following section shows the main methodologies, and the application is described in the Results and discussion section. The final section displays the main conclusions.

The Danjiangkou reservoir is a multipurpose reservoir in the Hanjiang River located to the west of Hubei province, China. The Danjiangkou reservoir basin (32.6°36′–33°48′N, 110°59′–111°49′E) has an area of 95,217 km², accounting for about 60% of the Hanjiang basin as shown in Figure 1. The annual mean inflow runoff at its dam site is about 36.81 billion m³, of which 61.4% concentrates during the flood season from July to October. The reservoir is also the water source of China's central line project of south-to-north water diversion, which benefits 50 million citizens suffering from water shortages (Liang et al. 2017).

The topography of the basin is very complex, of which the mountainous area accounts for 85% of the land area. The basin belongs to a sub-tropic monsoon climatic region with transitional climatic characteristics. The annual precipitation varies from 850 to 1062 mm and the mean discharge is about 1,250 m³/s (Li et al. 2009). The mean flood concentration time of the basin is about 3 days. With comprehensive benefits of flood control, hydropower generation and water supply, the 1–3 d lead time inflow forecasts have drawn special attention from the reservoir managers, which will directly influence the reservoir operation decision process.

Two types of data are used in this study, including precipitation and reservoir inflow records. The daily inflow data of the Danjiangkou reservoir from 2008 to 2013 (2192 days in total) were collected from the Water Conservancy Bureau of the Hanjiang River. The daily areal precipitation from 2008 to 2013 was derived from the China Gauge-based Daily Precipitation Analysis (CGDPA) database by grid-based averaging method. The CGDPA data can be downloaded from the meteorological data center of China Meteorological Administration (CMA, http://data.cma.cn) and have proved to be reliable for providing high quality daily accumulated rainfall amount ending at 00:00 UTC over the mainland of China (Xie et al. 2007; Shen et al. 2010; Sun et al. 2016).

The NWP data from 2008 to 2013 in the Danjiangkou reservoir basin were downloaded from the TIGGE database retrieval website at ECMWF (http://apps.ecmwf.int/datasets/data/tigge). A summary of the three TIGGE models is presented in Table 1. All the NWP data were spatially interpolated from the original NWP data automatically by the website server at a spatial resolution of 0.5° × 0.5°. The original time interval was 6 h and the daily precipitation predictions were obtained by summing the four successive predictions. As shown in Table 1, the ECMWF model had the most ensemble members, which were more than double of the NCEP and three times that of the CMA. The areal precipitation predictions of the Danjiangkou basin were calculated using a grid-based averaging method.

In this study, a probabilistic forecasting scheme was constructed to derive probabilistic inflow forecasts in the Danjiangkou reservoir (see Figure 2). Three artificial neural network (ANN) models were calibrated first, the input combinations of which were optimized with one of the state-of-the-art input selection methods, i.e. the Gamma Test. After that, ensemble forecasts of the Danjiangkou reservoir with three lead times were generated using both the antecedent flood and precipitation records and also the corrected TIGGE NWP data. The BMA method was applied to generate probabilistic forecasts with the ensemble forecasts. The probabilistic forecasts were evaluated and compared with the raw ensemble forecasts under a coherent set of evaluation criteria. The methods employed in this study are briefly addressed in the following.

The Gamma test (GT), introduced by Agalbjorn et al. (1997), is a data analysis technique for assessing the extent to which a given set of M data points can be modeled by an unknown smooth non-linear function. The Gamma statistic (Γ) is an estimator of the model output's variance that cannot be accounted for through a smooth data model. The most attracting feature of the GT method is that it is sufficient for input selection without knowing the input–output system structure. Recent studies revealed that the GT method is an effective way of identifying ANN input and can thus reduce the input dimensions as well as produce precise output of ANNs (Moghaddamnia et al. 2009; Noori et al. 2011; Chang et al. 2013, 2014). Therefore, the GT method is used to determine the combinations of antecedent inflow and precipitation for the ANN models in this study. A brief introduction of conducting GT follows:

ANN is an intelligent algorithm widely used in pattern classification, clustering, function approximation, forecasting, image completion and many complex problems in the real world (Hajmeer et al. 2000). With the strong power of describing complex systems, the ANN model has been widely used to simulate various hydrologic processes and previous researchers reported that the ANN model performs quite well on flood forecasting (Chiang et al. 2007; Kan et al. 2015; Nanda et al. 2016; Ba et al. 2017).

Previous research literature shows that it is essential to correct the TIGGE data in order to use them more effectively due to the following reasons: (1) the accuracy of the raw TIGGE data are still limited nowadays and thus unsuitable for direct application to hydrological forecasting, despite the fact that the NWP techniques have gained significant improvements; (2) the spread of the raw TIGGE data may be unreliable since the uncertainty range obtained from the ensemble spread may not contain statistically consistent observations; and (3) the raw spatiotemporal resolution of the TIGGE data may not satisfy the forecasting models (Tao et al. 2014). A simple while effective correction method is applied here to preprocess the TIGGE ensemble NWP (Peng et al. 2015):

1. When the precipitation prediction result does not exceed the heavy rain threshold, the prediction value is adjusted by the following equation:

   

   (8)

   where P_f indicates the precipitation prediction, and indicates the mean value of the historic predictions and observations, respectively, and P_crt indicates the corrected prediction.
2. When the precipitation prediction indicates an intense rain event, the prediction value is adjusted by the following equation:

   

   (9)

   where P_mid indicates the middle value of the historic observed intense rain event, which is 40.4 mm/d in this study basin. The method is used to correct the systemic bias of the TIGGE NWP and meanwhile give sufficient attention to severe rain events (Peng et al. 2015).

The NSE ranges from –∞ to 1 and is positive oriented (Nash & Sutcliffe 1970). The MAE and the RMSE are widely regarded as standard metrics for forecasting errors in the fields of hydrology, for which low values indicate a good forecasting skill (Gneiting et al. 2007; Chai & Draxler 2014).

When the forecast G is deterministic, the CRPS will reduce to the MAE (Gneiting et al. 2007). This makes it convenient for making comparisons between deterministic and probabilistic forecasts.

The evaluation metrics for the calibrated ANN models are listed in Table 3 and the simulation results during calibration and validation periods are displayed in Figure 4. It can be seen that the ANN models have quite good performances and can fit the observed inflow series very well. The ANN simulated inflows with 1 and 2 d lead times are relatively better than that of 3 d lead time, which are obviously overestimated during the calibration period in Figure 4(c). All the NSE values during calibration period are larger than 0.85 and the MAE values and RMSE values are very low. The validation periods appear less satisfied than the calibration periods under the criteria of NSE, MAE and RMSE. Hence, the three ANN models are used to generate ensemble inflow forecasts.

As mentioned above, the raw TIGGE NWP must be corrected before applying to hydrometeorological purposes. Figure 5 shows the comparisons of NSE, MAE and RMSE values of the raw TIGGE data and the corrected TIGGE data. For the raw TIGGE data, it can be seen from Figure 5 that the accuracy of precipitation prediction declines with lead time increase. Among the three TIGGE models, the ECMWF performs the best. The TIGGE data from the CMA generally performs worst with the smallest NSE values and the largest MAE and RMSE values. By the correction method introduced above under ‘NWP data correction’, significant improvements are observed for these three metrics above. The systematic errors are sufficiently removed since the corrected TIGGE data are more approximate to the precipitation observations. The corrected TIGGE data are used as ANN inputs to generate ensemble flood forecasts together with the antecedent flood discharge of the Danjiangkou reservoir.

By feeding the corrected TIGGE data into ANN models with the antecedent precipitation and discharge records, a total of 87 ensemble inflow forecasts (the raw ensemble inflow forecasts) are obtained in accordance with the TIGGE NWP member size for each lead time. Figure 6 displays the evaluation results of the raw ensemble inflow forecasts with box-whisker plots. The upper and lower ends of the whiskers are the estimated 95% and 5% quantiles; the upper and lower edges of the boxes are the estimated 75% and 25% quantiles, respectively. The center lines are the median and the crosses outside the whiskers are outliers. It can be seen from Figure 6 that the widths of the evaluation metric intervals apparently increase with the lead time, which reveals that the discrepancy between the inflow forecasts is more significant as the lead time increases. This phenomenon indicates that the forecasting uncertainty cannot be neglected for middle- and long-term hydrologic forecasting. The reservoir managers and decision makers appreciate forecasting results with more accuracy and less uncertainty. It is very difficult to make decisions with the ensemble inflow forecasts that are quite different from each other. This difficulty for choosing a ‘best’ forecasting member seems more difficult for longer lead times since the uncertainty is more significant (the results are more dispersive), and in fact no single forecasting member can always have the best performance, which is widely addressed by the previous research studies (Cloke & Pappenberger 2009; He et al. 2009; He et al. 2010; Alfieri et al. 2013). To better support the decision making, a useful approach is to generate probabilistic inflow forecasts based on the ensemble results using post-processing methods (Cloke & Pappenberger 2009; Gneiting & Katzfuss 2014).

We use the BMA method to generate probabilistic inflow forecasts from the raw ensemble inflow forecasts, the calibration period (2008–2011) and the validation period (2012–2013) are consistent with the ANN models. For calculation convenience, the conditional distribution of the ensemble members is supposed to follow Gaussian distribution (Duan et al. 2007). The Box-Cox transformation is performed on the observed inflow series and the ensemble inflow forecasts to make them more Gaussian distributed. The transformed and original series are plotted in Figure 7. Since the ensemble member number is large, only one inflow forecast member is chosen arbitrarily to display the Box-Cox transformation performance. It can be seen clearly that the original inflow series are non-Gaussian distributed. After the Box-Cox transformation, the observed and ensemble forecasting inflow series are very close to the theoretical Gaussian distribution probability curve, which proves that the transformed series can satisfy the Gaussian assumption well.

Figure 8 shows the PIT histogram of the raw ensemble forecasts and the BMA probabilistic forecasts during the validation period. The CD values are calculated by Equation (19) and shown in the figures. The reference line of a perfectly flatted histogram is also shown on each plot, which is a function of the number of the bins and the sample size (Bourdin et al. 2014). Figure 8 clearly illustrates that the raw ensemble forecasts are highly unreliable. The PIT values are seldom distributed at the center of the probabilistic density, while the first and last bins almost accounted for the whole ensemble. After probabilistic calibration using the BMA method, the reliability of the probabilistic forecasts is effectively improved. Many more PIT values are distributed at the center of the probabilistic density. The PIT histograms of the BMA probabilistic forecasts are more flat compared with the raw ensemble ones, although the first and last bins still have more PITs. The metric CD is more intuitional than the PIT histogram. For each lead time, the CD value of the BMA probabilistic forecasts is much smaller than that of the raw ensemble forecasts. Results indicate that the BMA probabilistic inflow forecasts are more reliable than the raw ensemble ones.

Figure 9 shows the CRPS and IGN values of the raw ensemble forecasts and the BMA probabilistic forecasts for each lead time during the validation period. The CRPS can represent both the reliability and the sharpness of the probabilistic forecasts and the IGN is a metric that illustrates the ability of the probabilistic forecasts' concentration at the right area. It can be seen from Figure 9 that the CRPS values of the BMA probabilistic inflow forecasts are lower than the raw ensemble ones. This indicates that the BMA probabilistic inflow forecasts have better performance than the raw ensemble inflow forecasts for all three lead times. It is also noted that this advantage becomes less significant when the lead time increases. The results of the IGN values in Figure 9 are in agreement with the CRPS values.

Another finding in Figure 9 is that the IGN values of the raw ensemble inflow forecasts show a descending trend with the lead time, indicating the raw ensemble inflow forecasts with longer lead time can better contain the correct areas. This is caused by the ensemble generation method which couples TIGGE NWP with ANN models. With the lead time increases, more NWP data are used as model inputs and additional uncertainty are introduced into the ensemble forecasting results. As shown in Figure 6, the metrics with a 3 d lead time have the widest spreads, which indicates that the 3 d members are less sharp or consistent than the 1 and 2 d results. The 1 d raw ensemble inflow forecasts are the sharpest and have the highest IGN, which indicates that the 1 d raw ensemble forecasts are sharp but less reliable. Results show the raw ensemble forecasts fail to balance the sharpness and the reliability. After applying the BMA method, it is observed that the trends of the two metrics (CRPS and IGN) are consistent, i.e. the probabilistic inflow forecasts with longer lead time are less reliable and sharp.

The deterministic inflow forecasts derived from the raw ensemble forecasts and the BMA probabilistic forecasts are also evaluated since the deterministic results occupy an important position in practice. The results in Table 3 show that the BMA deterministic forecasts have better performances than the raw ensemble ones for all three lead times under the criteria of NSE, MAE and RMSE. The main reason for these improvements is that the BMA is essentially a bias correction method, which gives larger weights to the better ensemble members and vice versa (Duan et al. 2007). Given an overall conclusion, both the evaluation results of probabilistic forecasts and deterministic forecasts indicate that the scheme of post-processing the ensemble forecasts generated using TIGGE NWP data by the BMA method offers a useful approach for probabilistic forecasts in the Danjiangkou reservoir basin.

Nowadays, more and more attention is paid to obtaining timely and accurate flood forecasts. The development of numerical weather prediction (NWP) techniques has contributed to this goal by providing ensemble precipitation predictions, which take longer lead times and can describe the uncertainty of the meteorological system. In this study, the NWP data retrieved from three TIGGE models were used as the inputs of ANN models to forecast reservoir inflows. The ensemble inflow forecasts were then post-processed to generate probabilistic forecasts using the BMA method. The main conclusions of this study are summarized as follows:

1. The raw TIGGE NWP data are biased and must be corrected before implementing the hydrologic model. The correction method applied in this study can effectively correct the TIGGE NWP under the criteria of NSE, MAE and RMSE in the Danjiangkou reservoir basin.

2. The raw ensemble inflow forecasts are unreliable. The ensemble spread is very narrow for 1 d results while much wider for the 2 and 3 d results. The PIT histograms of the raw ensembles show significant U-shapes with almost all the PIT values being located at the first and the last bins.

3. The probabilistic forecasts derived from the raw ensemble forecasts by using the BMA method are significantly improved. The PIT histograms of the probabilistic forecasts are flatter as compared with the raw ensemble ones, and the CD, IGN and CRPS values of the probabilistic forecasts are also lower than those of the raw ensemble forecasts for all three lead times. Results show that the BMA can provide a useful approach to generate probabilistic forecasts from the TIGGE ensemble inflow forecasts.

There remain some limitations in this study. The BMA parameters were optimized with the data of the calibration period but did not change over time. This may be improved in future by using a sliding window calibration method to optimize the parameters adaptively (Bourdin et al. 2014). Another limitation is that the Box-Cox transformation may be replaced by the Normal Quantile Transformation (NQT) to make the data more Gaussian distributed, which can better fulfill the Gaussian assumption. Besides, as discussed by many studies (Xiong et al. 2009; Li et al. 2010; Kasiviswanathan et al. 2013), the reliability and the sharpness of probabilistic forecasting are competitive, how to balance the two targets is an interesting question to be investigated in future.

This study was supported by the National Key Research and Development Plan of China (Grant No. 2016YFC0402206) and the National Natural Science Foundation of China (Grant No. 51539009, 51779279 and 91647106). Thanks are also given to the three anonymous reviewers for their great efforts to help improve this manuscript.