A robust real-time flood forecasting method based on error estimation for reservoirs

In conventional real-time flood forecasting, parameter estimation and real-time flow updating are mostly carried out based on the residuals between measured discharge and discharge simulated by hydrological models (Bao & Zhao 2014). However, in modern forecasting systems, the discharge and precipitation are generally measured automatically by telemetering devices (Chao et al. 2008), which have the advantages of 24-h online monitoring, high precision, and labor-saving compared to the traditional measurement. Even so, the observed data inevitably contain some extreme errors and gross errors, which are irregular, unpredictable, and difficult to identify, especially in heavy rain. These errors have negative effects on the development of forecasting model, calibration of model parameters, and flow updating in real-time flood forecasting. Although there are many statistical tests in traditional least square estimation which can detect gross errors (Chan & Zhou 2009), the least square estimations are only unbiased, consistent, and valid when the residuals of observations follow the normal distribution. However, there are few observed data that strictly follow the normal distribution in real cases. There are always some extreme errors and gross errors in observed data due to some objective factors like instruments. The least square estimations are not valid for cases following a contaminated distribution (Chan & Zhou 2009). That is because the statistics used are calculated by the least square. Hence, it is almost impossible to separate the extreme errors and gross errors from observed data in the error estimation. And some gross errors are difficult to detect by the traditional methods in complex systems or multidimensional regression fitting problems, especially the gross errors in highly automated real-time flood forecasting systems.

In order to solve this problem, a new method known as the robust estimation method has been developed. It can detect and deal with gross errors better than the least square estimation method. However, it was not until the 1960s that substantial development was made to the robust estimation method when Tukey (1960) proposed the error distribution. After the development, the method can reduce the effects of gross errors with weighting factors based on the distribution of residuals. This robust method is different from the conventional recursive least square method, in which the residuals are transformed non-linearly. Based on the robust theory, Zhou (1989) proposed the concept of equivalent weights to ameliorate the conventional least square algorithm. Then, Bao et al. (2003) introduced the robust estimation theory, which was well recognized in statistics, into hydrological forecasting. He pointed out that in applying robust estimation to hydrological forecasting, it is important to select a robustness function that is appropriate to the hydrological problem. Several widely used robust estimation methods, such as the Huber estimation method, the Tukey estimation method, and the IGG estimation method, were investigated and applied to the parameter estimation of the hydrological models (Bao et al. 2003). A recursive robust method for online estimation of the time-varying parameters of the auto-regressive updating model using a weighted least square method with forgetting factors was developed (Chao et al. 2008). Moreover, based on the telemetry measured rainfall distribution characteristics and the robust estimation theory, a three-stepwise updating method was developed (Zhao et al. 2010). Furthermore, in the application of parameter estimation for the Muskingum Model (Shen et al. 2016), the robust least square algorithm was shown to be efficient and was able to handle the gross errors and extreme errors.

Hitherto, the robust estimation method has been mainly used for the parameter estimation of hydrological models (Finsterle & Najita 1998; Cizek 2001; Bárdossy & Singh 2008; Tsanov et al. 2020; Krisnayanti et al. 2021) and flood risk management (Mens et al. 2011; Lamond & Penning-Rowsell 2014; Kerich 2020). It has rarely been applied to real-time flood forecasting. For reservoir inflow, it is usually not measured directly but calculated based on the observed water stage, the reservoir outflow, and the stage-storage curve (Chao et al. 2008). The errors in the water surface observation may greatly reduce the accuracy of real-time updating in flood forecasting.

In this study, in order to improve the accuracy of flood forecasting in reservoirs and preserve the stability of the forecasting system, a robust real-time flood forecasting method based on error distribution has been developed. This new method takes into account the contaminated distributions of outliers in the observed data and suppresses the outliers by a robust loss function. Furthermore, a fluctuation coefficient has been proposed, which can describe the fluctuation of reservoir inflow caused by gross errors. With the fluctuation coefficient, the relationship between the degree of fluctuation of a flood and the performance of the robust real-time flood forecasting method can be analyzed quantitatively. By applying the method to floods generated by an ideal model, the efficiency and robustness of the method has been assessed. Furthermore, by applying the method to 10 reservoir basins with different characteristics and conditions, the superiority and applicability of the method has been verified. Additionally, the results indicate that the performance of the proposed method is positively related to the fluctuation coefficient of floods.

In general, floods are often accompanied by strong winds, which lead to gross errors in the reservoir stage measurements. Since is calculated based on the reservoir storage-stage curve which is positive related, there is an amplification of error in the calculation from the reservoir stage measurement to the reservoir inflow. This amplification is positively correlated to the water surface area of the reservoir and negatively correlated to the time period. Therefore, the errors in the reservoir inflow usually display an unknown distribution with zero mean and a large variance.

In fact, the degree of reservoir inflow fluctuation varies from basin to basin. It can even vary from flood to flood. Floods during non-flooding seasons are relatively stable, because the positive and negative errors generally offset each other. However, during flooding seasons with strong winds, the errors in the reservoir stage observation are significantly larger. As the water surface area varies with the reservoir stage, the errors in the reservoir stage observation can cause fluctuations in the reservoir inflow, resulting in extreme errors and gross errors. Figure 1 shows a typical example of fluctuations in the reservoir inflow, which includes negative inflow.

Originating from the concept of robustness in statistics, the robust estimation method minimizes the influence of outliers on parameter estimation or error estimation as much as possible. With the remaining outliers, an appropriate estimation method is chosen so as to obtain an optimum parameter. There are two advantages in this approach. (1) When a hypothetical model is slightly different from an actual model, the performance of the estimation method is only slightly affected. There is stability in the estimation method. (2) When there are a number of outliers in the observed data, the estimated value is not severely affected. There is anti-interference in the estimation method.

The robustness estimation can be broadly classified into three types: (1) M-estimation (Chan & Zhou 2009), (2) L-estimation, and (3) R-estimation. The robust M-estimation is an extension of the classical maximum likelihood estimation. Due to the maturity of the theory and simple calculations, it is widely recognized and commonly used. Furthermore, it can be easily generalized to become a multidimensional parameter estimation method.

Equation (5) is the robust estimation criterion.

The effect of Equation (10) is to assign a smaller weight to a small portion of large residuals so that the impact of the outliers on the final estimates is less significant, while giving a unit weight to the bulk of small moderate residuals (Chao et al. 2008).

In this study, by introducing the robust estimation to flood forecast, a robust real-time flood forecasting method has been developed in which the unwanted errors are assigned with weighting factors according to the error distribution theory. The calculation procedure of the robust real-time flood forecasting method is as follows:

• 1.

  Use a hydrological model to calculate the reservoir inflow, known as Q_cal. In this study, the Xinanjiang model, which was developed in 1973 by Zhao (Zhao 1992; Zhao et al. 1995) and improved by the researchers of Hydrological Forecast Research Institute in Hohai University (Bao et al. 2014; Si et al. 2015), has been used. The model consists of four components: (1) evapotranspiration, which consists of a three-layer soil moisture structure, (2) saturation excess runoff mechanism, (3) slope flow concentration, which consists of linear reservoirs, and (4) stream flow concentration, which uses the Muskingum method.

• 2.

  On a piecewise basis, fit the observed reservoir inflow by quadratic curves to generate a relatively smooth discharge curve. The number of sample points for each segment, known as smooth hour, is determined by the characteristics of each flood. The principle of segmentation is that the smooth hour should be long enough to eliminate the zigzag fluctuations of the reservoir inflow while keeping the overall contour of the flood process, especially the flood peak.

• 3.
  Calculate the residuals of the measurements and update the jagged inflow with the following equations:

  

  (12)

  

  (13)

  where Q_o is the observed discharge, Q_r is the discharge after the robust correction, and Q_s is the smooth discharge determined by Step 2. In Equations (12) and (13), is calculated by Equations (10) and (11).
• 4.

  Re-fit the discharge after the robust correction, Q_r in Step 3, to generate a smooth robust discharge, known as Q_rs.

• 5.

  Based on the differences between Q_rs and the discharge Q_cal calculated by the hydrological model, use the recursive least square method to update the real-time flood.

In order to investigate the validity and effectiveness of the developed method, it has been applied to floods generated by a theoretical model and in 10 selected reservoirs.

Table 1 shows the average fluctuation coefficient and the performance of the developed method for the 10 floods with different error magnitudes generated synthetically in the ideal model.

Based on the results in Table 1, two primary conclusions can be drawn:

• 1.

  Since the fluctuation coefficient increases with increasing error magnitudes, the fluctuation coefficient can quantitatively describe the degree of fluctuation of floods.

• 2.

  For the floods with different error magnitudes generated synthetically in the ideal model, the robust mean square error is reduced significantly after robust updating. This result shows that the robust real-time flood forecasting method is efficient and steady. The method can, therefore, be applied to real-life reservoirs.

Figure 2 shows the residual reduction rates of the 10 floods with different error magnitudes generated synthetically in the ideal model. Figure 3 shows the relationship between the average residual reduction rate and the average fluctuation coefficient of the 10 floods. The results show that there is a linear relationship between the residual reduction rate and the fluctuation coefficient, and they are positively related.

As a way to further investigate the efficiency and robustness of the developed method, 10 reservoirs of different characteristics have been selected. The selected reservoirs are distributed over different river systems in humid regions. Furthermore, the area, climatic conditions, and storage capacity of these reservoirs are very different from each other. The characteristics of the selected reservoirs are shown in Table 2, and the locations of the study areas are shown in Figure 4.

Table 3 shows the results of the simulations by the robust real-time flood forecasting method for the floods in the Qingshitan Reservoir. The results of the simulations for the floods in the other reservoirs are similar. The results show that the robust mean square errors of all the floods are significantly reduced after robust updating. From Table 3, it can also be seen that the simulation error of the runoff (e_sx and e_kc) is very small. This is an indication that the simulation results satisfy the water balance equation. Hence, the simulations by the robust real-time flood forecasting method are accurate.

Table 4 shows the average results of the method for the floods in the 10 selected reservoirs. It can be seen that the mean robust mean square errors of all the floods in different reservoirs are reduced after robust updating, and the average residual reduction rate is 20.97%. For a single flood, the reduction rate can reach 60% (e.g. Flood 850603 of the Lushui Reservoir). Moreover, the observations with significant robust effects are mostly close to the flood peaks, and for the floods with rapid confluence and precipitous fluctuation, the residual reduction rate is larger. This is because close to the flood peak, the fluctuation of the water surface is greater. Since the fluctuation phenomenon near the flood peak is more severe, the observed discharge data around the flood peak are more likely to contain outliers, especially for floods with rapid confluence and precipitous fluctuation. Based on the above analysis, the robust real-time flood forecasting method is rational and effective.

Figure 5 shows the relationship between the residual reduction rate and the fluctuation coefficient of the floods in the Qingshitan Reservoir. Figure 6 shows the statistical results for the floods in 10 selected reservoirs. From Figure 5, it can be seen that the larger the fluctuation coefficient, the higher the residual reduction rate. In other words, the robust effect is more significant. This phenomenon is also observed in the other nine reservoirs. Furthermore, from Figure 6, it can be seen that for different reservoirs, the robust performance of the method is positively correlated to the average fluctuation coefficient of floods in the reservoir. This is an indication that the method is robust.

Comparing Figures 4 and 5, three mentionable differences can be seen. First, the average residual reduction rates of the 10 floods with different error magnitudes generated by the ideal model are between 40 and 75%, while the residual reduction rates of the floods in the Qingshitan Reservoir are all smaller than 40%, illustrating that the performance of the method in an ideal model is overall better than in real cases. Second, the slope of the trend line in Figure 4 (i.e. 53.17) is smaller than that in Figure 5 (i.e. 75.62), suggesting that the method is more robust in the ideal model than in real cases. Third, the correlation coefficient of the trend line in Figure 4 (i.e. 0.973) is much larger than that in Figure 5 (i.e. 0.76). This is an indication that the linear correlation between the residual reduction rate and the fluctuation coefficient for the floods generated by the ideal model is more significant than that for the floods in the real case. Underlying the differences is that the true values, the errors, and the error structure are known in the ideal model, so the gross errors are easier to detect and locate as compared to the floods in real cases. Hence, the method performs better in the ideal model than that in real cases.

In addition, comparing Figures 5 and 6, it can be seen that the slope of the trend line in Figure 5 is steeper than that in Figure 6. In fact, this observation applies to the floods in the other nine reservoirs. This is an indication that the relationship between the fluctuation coefficient and the robust effect is more significant in a single reservoir than in a group of 10 reservoirs. This is because for the group of 10 reservoirs, the relationship between the residual reduction rate and the fluctuation coefficient is an average. Hence, the relationship is less significant.

Furthermore, the slope of the trend line between the mean residual reduction rate, E_v, and the mean fluctuation coefficient, α, of all 10 selected reservoirs have been calculated. Figure 7 shows the relationship between the slope of E_v and α and the average fluctuation coefficient of 10 selected reservoirs. The results show that the slope of E_v and α is inversely proportional to the average fluctuation coefficient of reservoirs. That is because, for reservoirs with a large average fluctuation coefficient, the number and magnitude of the outliers in the observed flood data is large. As the residual reduction rate of different floods is generally large, the robustness of the method remains steady for different floods. On the other hand, for reservoirs with small average fluctuation coefficient, the frequency and magnitude of the outliers is small. So, not all of the observed flood data contain outliers. The residual reduction rate for the observed flood data with outliers is large, while the residual reduction rate for the observed flood data with little or no outliers is small. Therefore, the robustness of the method on different floods changes greatly, resulting in a large slope of E_v and α, as shown in Figure 7.

Many methods and techniques have been developed to improve the accuracy of real-time flood forecasting in recent years. Most of the methods are focused on real-time correction, which can be classified into terminal error correction and process error correction. For example, the dynamic system response curve (DSRC) method, a process error correction method, has been developed to improve flood forecasting. There have been many achievements and applications of the DSRC method in the hydrological model and real-time flood forecasting in recent years (Si et al. 2013, 2015; Liang et al. 2021). Most of these process error correction methods rely on the accuracy of observed data. So, when they are applied in reservoirs, the accuracy is limited. However, the method proposed in our study is a terminal error correction method that can detect and correct the gross errors in data directly based on the actual error distribution of the observations. So, it is robust and applicable enough for any system with random error and outliers.

Due to the errors in the water level measurements and the inverse calculation to obtain the reservoir inflow, the reservoir inflow data inevitably contain some outliers and gross errors, resulting in a jagged flood hydrograph. In this study, by introducing the robustness functions into real-time flood updating to limit the influence of gross errors on the forecasting results, a new robust real-time forecasting method has been developed. Furthermore, the fluctuation coefficient has been proposed, which can describe the irregular fluctuation quantitatively. The performance of the developed robust real-time flood forecasting method has been assessed by applying it to the floods generated by the ideal model and to floods in 10 selected reservoirs. The application results for both synthetic and real floods show that the robust real-time flood forecasting method is efficient and universally applicable. Even for flood data with outliers, the flood forecasting using the developed method is stable and accurate. Moreover, the robustness of the method is proportional to the fluctuation coefficient. In other words, the more severe the fluctuation of the flood hydrograph, the more robust is the method. And the robust real-time flood forecasting method can be a flexible method for different systems with disparate error structures through the selection of diverse robustness functions. In addition, the method is merely applied in humid regions and only utilized in conjunction with the Xinanjiang model in this study. The relationship between the fluctuation coefficient and the robust effect of the method in other regions of different characteristics as well as in other hydrological models, such as in arid areas, remains to be further verified.

This study is supported by the Postgraduate Research & Practice Innovation Program of Jiangsu Province (Grant No. SJKY19_0466), the Fundamental Research Funds for the Central Universities (Grant No. 2019B72514), the National Key R&D Program of China (Grant No. 2016YFC0402703), the fundamental research funds for central public welfare research institutes (Grant No. HKY-JBYW-2017-12), Open Fund of Key Laboratory for Technology in Rural Water Management of Zhejiang Province (Grant No. ZJWEU-RWM-20200202B), and the National Natural Science Foundation of China (Grant Nos. 51709077, 41371048, 51479062, and 51709076).

The data that support the findings of this study are available from the hydrological stations in the study basins. Restrictions apply to the availability of these data, which were used under license for this study. Data are available from the corresponding author with the permission of the hydrological stations in the study basins.

All authors declare no conflicts of interest.

All relevant data are included in the paper or its Supplementary Information.