Augmentation of an artificial neural network and modified stochastic dynamic programing model for optimal release policy

Development of a decision-making model for reservoir operation that depicts field-scale conditions is a challenging task, and a variety of different modeling approaches exist for this purpose. Many researchers have published comprehensive reviews of optimization and simulation techniques related to reservoir operations and modeling (Yeh 1985; Labadie 2004; Fayaed et al. 2013a, b). Chandramouli & Deka (2005) and Chandramouli & Raman (2001) developed new approaches with the decision support model (DSM) based on trained neural network models that use real-time data from previous time periods to decide on operating policies. The DSM that was developed based on an artificial neural network (ANN) outperforms the regression-based approach. Chaves & Kojiri (2007a, b) introduced a new approach for system optimization and operation known as the stochastic fuzzy neural network, which can be defined as a neuro-fuzzy system that is stochastically trained (optimized) by a genetic algorithm (GA) model to represent the system operational strategy. Moreover, to address the imprecision originated by the discretization of inflow intervals (events) in calculating the transition probabilities, the researchers applied a method based on the conditional probability of a fuzzy event.

Uncertainties associated with random properties commonly exist in a reservoir operation system (e.g. precipitation, stream inflow, and water demands); moreover, these uncertainties are further compounded by ambiguity and vagueness from the relevant hydrologic information obtained (Jing & Chen 2011). Previously, a number of stochastic dynamic programing (SDP) methods were explored to reflect the dynamic complexities and random uncertainties of the reservoir operation system (He et al. 2010; Housh et al. 2013). The current work considers the ability of ANN to mimic the nonlinearity features of the reservoir system and shows that the ANN model's capabilities are suitable for hydrological processes. The ANN is a useful tool for resolving complex problems than can be addressed using other traditional models (regression analysis (RA)). In this work, the results obtained showed that the ANN provides more accurate and reliable results than other methods, especially in the simulation result obtained on the relationships between the elevation–storage and storage–surface area capacities of the reservoir. Chaves & Chang (2008) proposed a novel intelligent reservoir operation system based on an evolved ANN. The reference to evolution indicates that the parameters of the ANN model are identified using the GA evolutionary optimization technique. Accordingly, the ANN model should represent the operational strategies of reservoir operation. This system is capable of successfully and simultaneously handling various decision variables and provides reasonable and suitable decisions.

Adeloye & Munari (2006) developed an optimization model integrated with ANN and regression models. The results showed that the latter are marginally better; however, given that the regression models require the over-year capacity to be known a priori, the ANN models are more generic and should be preferred.

An important controversy in the literature of stochastic reservoir operation and stream flow modeling concerns the appropriate statistical assumptions for the inflow sequence (Philbrick & Kitanidis 2001; Ponnambalam et al. 2003; Liu et al. 2006; Chaves & Chang 2008). The appropriateness of a stochastic assumption for this inflow sequence assumption will depend on the interval between time steps.

These remarks are cogent because many authors have adopted the independent assumption. For computational reasons (compared with the independence assumption, the first-order Markov process assumption does not add more complexity and is consistent with the solution process of the SDP formulation), the first-order Markov process description of inflows has been adopted in nearly all of the SDP formulations in statistical studies that tested the Markov assumption on rivers (Mousavi et al. 2005; Panagoulia 2006; Mousavi et al. 2007; Rani & Moreira 2010; Safavi et al. 2013).

In this situation, a first-order Markov process is not appropriate. If the correlations among inflows beyond lag one are strong, the occurrence of current inflows can be more precisely conditioned on previous inflows. In other words, the current inflow can be more precisely modeled using not only the inflow in the previous time step but also using the inflows in the preceding time steps (higher lags). The transition probability of the current inflow should be conditioned on additional previous inflows, which is a more precise description of reality. However, this approach yields a mathematical formulation that is impractical to solve, because these previous inflows must be included in the state variable set. These difficulties stem from the structural limitations of the SDP in which the addition of each new state variable forces a significant increase in the number of system states to be evaluated. Therefore, in SDP reservoir operation modeling, the number of state variables rarely exceeds two (one storage variable and one hydrological variable) (Raman & Chandramouli 1996; Jain et al. 1999; Moeini et al. 2011; Kumar et al. 2013).

An innovative approach to tackling the dimensionality problem treats the inflow sequences as inflow patterns, which are represented in the SDP formulation as a single state variable. The resultant model may be referred to as an inflow pattern stochastic dynamic programing model.

However, the efficiency and successful operation of a reservoir depends largely on the reservoir simulation system. The determination of the reservoir simulation approach must therefore be given special attention. Different methods can be used to determine the simulation, and most of them make use of statistical analysis. However, the problem with most of these methods is that they cannot easily handle the phenomenon of nonlinearity processes that occur in the reservoir (Celeste & Billib 2009; Fayaed et al. 2011). Simulation models associated with reservoir operation are usually based on a mass balance equation, which represents the hydrological behavior of reservoir systems using inflows and other operating conditions (Wang et al. 2010; Wu & Chen 2013).

This study examines the integrated ANN as a reservoir simulation component and the modified stochastic dynamic programing model (MSDP) as an optimization method in determining optimal reservoir operation. This knowledge will assist in obtaining more accurate values for optimal reservoir policy, which will act as a foundation for the case study. In addition, this work also reviews the application of the MSDP model together with other simulation models and compares the effectiveness of the different results. This work provides an understanding of better performance that will be reflected in enhanced reservoir release and an effective distribution system, which are the results of a more accurate reservoir simulation.

This study considers the Sg Langat dam as a case study. This Malaysian dam is located geographically at latitude 3° 12′ 43.07″ north of the equator and longitude 101° 53′ 39.28″ east of the prime meridian in Kuala Lumpur. The dam was built and completed in 1979 for the purpose of water supply. The Sg Langat dam has an earth embankment made of 2.5 million cubic meters (MCM) of earth, and the crest elevation is 223.72 m with a maximum height of 61 m. The primary spillway is gated, and the crest elevation is 220.96 m. The reservoir catchment area is 41 km², as shown in Figure 1. The dam controls and regulates the volume of water released into the Langat river during the dry season to ensure that adequate water is available at the Sg Langat water treatment plant, located approximately 14 miles downstream of the dam. This dam plays a highly significant role in the neighborhood of Kajang, Bangi, because it is a major supplier of drinking water to the residents. The dam's (reservoir) capacity is 37,480.0 million L. The relationships among water surface elevation, storage volume, and water surface area constitute fundamental information that characterize the reservoir and are required in modeling studies. The elevation versus area relationship is used in evaporation computations, because evaporation volumes are a function of evaporation rates and water surface area. The elevation versus storage relationship is required to determine the head pressure in hydroelectric power computations. Storage/elevation relationships are required to relate storage capacities to elevation-based operation rule curves. The results from statistical analysis of 15 years of historical data pertaining to the Sg Langat reservoir are presented in Table 1. Table 1 shows that the average annual inflow is 46.78 × 10⁶ m³ with a deviation of 19.77 × 10⁶ m³. The average maximum inflow occurs during the month of September, and the average minimum occurs during August. The maximum and minimum inflows realized in each month are also depicted in Table 1.

The hypothesis of regression and correlation is a classical framework used to describe relationships between two or more variables. Of course, the relationship may not necessarily be physical in nature and might also involve methods for modeling and analyzing several variables if the focus is on the relationship between a dependent variable and one or more independent variables. This analysis further explains how the typical value of the dependent variable changes when any one of the independent variables is varied and the other independent variables are held fixed. Determination of the relationships among elevation, storage, and surface area is one example of the application of RA in the science of hydrology. In investigating the relationship between two variables x and y, the realizations of the random pairs (x_i, y_i) are plotted as a scatter plot, which illustrates the relationship between the two variables.

The set of all available data is separated in two disjoint sets: the training set and test set. The test set is not involved in the learning or training phase of the networks and is used to evaluate the performances of the models. In this study, 100 inputs of reservoir elevations (first 50 and last 50 inputs) were selected for calibration of the ANN model, whereas the remaining 19 inputs (in between) were used to verify or test the performance of the model. In this particular study, the structure of the ANN model is designed based on Figure 2. The structure of the elevation-area model is the same as that of the elevation-storage model, i.e. both models have the same structures. For additional details on the structure of the ANN simulation model for this reservoir, see Fayaed et al. (2013b).

During integration of the conventional stochastic dynamic programing with artificial neural network (CSDP-ANN) model, the results of the simulation and optimization model were more reliable and robust than those of the CSDP-RA, but the disadvantage of the first model is that it is computationally time consuming, which leads us to look for a modification for the CSDP-ANN model to tackle this problem. The MSDP-ANN procedure is shown and presented in Figure 3.

In conventional dynamic programing (DP), the optimal storage carry-over corresponding to a current storage s_t is determined via an exhaustive search approach, and to find for each s_t, every s_t+1 value should be searched (Figure 4, left), i.e. if s_t is discretized into n_t intervals and s_t+1 into n_t+1 intervals (equal interval length), as shown in Figure 4, then to determine the optimal for p, the n_t+1 discretized s_t+1 intervals should all be tested.

If the single-period utility function is concave, according to the monotonic relationship between s_t and ⁠, if we know the optimal state q (at stage t + 1) corresponding to p (at stage t), then to search for the optimal for p + 1, we need only to test two s_t+1 values at the states of q and q + 1 (Figure 4, right). Thus, by applying the monotonic relationship, the computation for determining for s_t can be potentially reduced from n_t+1 to 2. An improved DP algorithm is proposed (as shown in Figure 5), which includes the following steps. (1) Start: discretize s_t and s_t+1 into intervals of equal length and number them in ascending order, i.e. 1 to n_t and 1 to n_t+1, respectively. (2) Initialization: setting p = 1 (the minimum discretized s_t), search from 1 to n_t+1 to find its corresponding optimal ⁠, and initialize q = s_t+1*⁠. (3) Computation: setting p = p + 1, search between q and q + 1, and use the better of the two to update corresponding to p. (4) Repeat step 3 until p = n_t. Steps 1–4 are designed for DP computation at stage t.

Using recursive computation from T to 1, the T-stage reservoir optimization operation problem can be solved using Equations (1) and (2). As illustrated in Figure 4, when n = n_t = n_t+1, the computational complexity of the improved algorithm is n + 2*(n–1) = 3*n–2, whereas that of the conventional algorithm is n², i.e. each of the s_t+1 states (n) should be tested with each of the s_t states (n). Thus, for a large number of state discretizations (n), 3*n–2 < <n², and the computational order of the improved algorithm can be considerably reduced.

If assuming a partially concave dependence relationship between and S_t+1 (i.e. ⁠), given that the state transition probability P(Q_t+1/Q_t) is positive, the partial dependence relationship between FF_t+1(S_t+1) (the weighted sum of ⁠, Equation (10)) and s_t+1 is concave (i.e. ⁠). Thus, with a fixed Q_t (Equation (11)), F_t and FF_t+1(S_t+1)│Q_t are concave functions, and an assumption of partially concave dependence relationship between and s_t+1 leads to the same relationship between and S_t. Hence, if assuming that F_t (r_t) is concave, following the procedures described in the ‘monotonicity in reservoir operation analysis’ section, we can conclude that: (1) F_T–1(S_T–1, Q_T–1), … , F₁(S₁,Q₁) are partially concave functions of S_T–1, … , S₁, respectively; and (2) with a fixed Q_t, a monotonic relationship exists between S_t and (and also ⁠). If applying the monotonic relationship to improving the SDP, the procedures are shown in Figure 6 with two steps. Step 1: for a fixed value of Q_t, run the modified DP (Figures 4 and 5) to search for corresponding to each of the possible values of S_t. Step 2: update Q_t and repeat step 1 until all possible Q_t values are tested and corresponding to all (S_t, Q_t) combinations is identified. Using the modified SDP in the backward recursive formulation, the multistage stochastic reservoir operation problem can be solved.

In this work, integration between ANN and CSDP is applied to derive the reservoir operation policy for the Sg Langat dam. The nonlinearity of natural physical processes causes a major problem in determining the simulation of reservoir systems, especially if using the conventional linear methods, e.g. RA method. To tackle this problem, a nonlinear computational method, i.e. the ANN, is introduced as a simulation system.

The model implementation procedure is applied first, and the operating storage volume (level) and stream inflows are discretized. Next, the SDP solution approach begins by initializing the values of the objective function at the last stage (a period in the future) to zero for each combination of state variables. The reservoir system is simulated using the ANN. Afterwards, the process continues backwards along the temporal stages. Each iteration consists of T stages that complete one annual cycle. The algorithm finds the end-of-period storage levels for each combination of the discretized beginning-of-period storage level and the average inflow of the current period. The behavior of the policy convergence after several iterations is due to the characteristics of the Markov transition probability matrix incorporated into the recursive equation.

It should be noted that the reliability, resilience, and vulnerability value ranges between 0 and 1. For the reliability index, a value is close to one means that the model is effective, whereas the model is effective in the case in which the value of resilience and vulnerability is close to zero.

This section presents the results of applying the models to the case study. A 15-year time series of stream flow (1996–2010) is used for evaluation of the model for water allocation from the Sg Langat reservoir according to the water treatment plant demands. In all models, the performance indices are used to evaluate the results of the models. These indices show how often the system does not fail (reliability), how quickly the system returns to a satisfactory state once a failure has occurred (resiliency), and the significance of the consequences of failure (vulnerability). With respect to the problem statement, two different types of state variables (Q_t or Q_t–1) were used when each model was tested (CSDP-RA and CSDP-ANN). From these tests, four submodels were obtained and are given in Table 3.

However, the results from these models were not as accurate compared with the results of other methods used in this study.

To generate the operating policies for reservoirs, a CSDP-based model with the RA was developed. The results of the CSDP-RA model with Q_t and Q_t–1 as the state variables are presented in Table 4. In this table, the reliability, resiliency, and vulnerability (used as the performance indices for supply to the water demand at the downstream section of the reservoir) for a 15-year simulation period were 0.699%, 0.302%, and 0.263 MCM, respectively, when Q_t was used as state variable (Model 1). However, when the state variable was Q_t–1 (Model 2), the performance indices were 0.678%, 0.289%, and 0.277 MCM, respectively, when equal numbers of inflow and reservoir storage discretization were considered. The results show that in using Model 1, the reliability of supplying the demands improved by 3.1%. The implication for this result is that out of the 175 months in the study period, the total number of nonsupplied demand months (unsatisfactory states) is 52 months using Model 1 but only 56 months using Model 2. In addition, more robust and reliable results were obtained. As given in Table 4, the resilience in Model 1 can produce a gain of 4.4% compared with that of Model 2. The vulnerability values, i.e. the expected values of nonsupplied water demand, were approximately 0.263 and 0.277 MCM for Models 1 and 2, respectively. The penalty function was used as a measure of the performance of the reservoir system and also used to identify the decision that makes the greatest contribution to the operation of the reservoir. The objective is to release the amount of water that is closest to the target demand, and thus, the operational performance also can be measured in terms of the deviation of the release decision from its target demand. The penalty function attempts to minimize the functional value and forces the release to satisfy the target demand. From Table 4, the penalty values for Models 1 and 2 are 1.59 × 10^–5 and 1.67 × 10^–5, respectively. This result shows that the penalty value increased by 5% for Model 1 compared with that of Model 2.

In this model, integration between the ANN and CSDP was used to derive the reservoir operation policy in Sg Langat. To evaluate the performance of the CSDP-ANN model, the results of the performance indices are determined and compared. The reliability, resiliency, and vulnerability of supplying the water demand at the downstream section of the reservoir are 0.76%, 0.327%, and 0.235 MCM, respectively, for Model 3, whereas the values for the same parameters are 0.734%, 0.310%, and 0.244 MCM, respectively, for Model 4, considering the same numbers of inflow and reservoir storage discretization. The results of the performance indices in supplying the demand to each sector are presented in Table 5. The results show that Model 3 improved the reliability of supply by 3.5%, which means that the total number of nonsupplied demand months (unsatisfactory states) is 47 months when Q_t–1 was the state variable (Model 4) and approximately 42 months when Q_t was the state variable (Model 3). The resilience in Model 3 shows a gain of 5.4% compared with that of Model 4. The vulnerability values are approximately 0.235 and 0.244 MCM, respectively, in Models 3 and 4. The penalty value in Model 3 is much better than that of Model 4, and its gain was 4.5%; this result demonstrates that Model 3 is more reliable and robust in deriving the optimal operation policy for the Sg Langat reservoir. The number of storage is 60, and from Table 6, the computation time at this storage using the CSDP-ANN model was 365.5 minutes while it was 18 minutes when the MSDP-ANN model was used.

The CSDP-RA and MSDP-ANN were applied to generate reservoir operation policies. The results from the two models are compared in this section, as shown in Figures 7 and 8. The cumulative penalty values for each of the four models are presented in Figure 7. It can be observed that Model 3 has the lowest penalty value. In addition, from the data shown in Figure 8, Model 3 is more reliable, more resilient, and less vulnerable than the other models. To compare the accuracies of CSDP-RA and MSDP-ANN models, the state variable Q_t was fixed. Comparison between Models 1 and 3 showed that the latter increased the reliability and the resilience of the system by 8.7% and 8.2%, respectively, which means that the total number of nonsupplied demand months (unsatisfactory states) is 53 months in Model 1 and approximately 42 months in Model 3. The nonlinearity of natural physical processes constitutes a major barrier to successful simulation of reservoir systems, especially if using such conventional linear methods as the RA method. To surmount this difficulty, the ANN nonlinear computational method is introduced for simulation of the system. The results of this comparison highlight the superiority of using ANN as the simulation model and Q_t as the state variable in the optimization model. The result of applying this reservoir operation system to each month is a three-pronged relationship among storage number, monthly inflow, and final storage number. This relationship represents the steady-state policy for each prospective month. In Figure 9, February was taken as an example to illustrate the shape of the relationship.

From the above discussions, it can be concluded that the best model for deriving the optimal operation policy is MSDP-ANN with Q_t as the state variable (Model 3). In this study, experiments were conducted to investigate the behavior of the system under different demand scenarios with increments of 5, 10, 15, 20, and 25% in the demand target, as given in Table 7. The results showed that an increase in the demand led to an increase in the vulnerability and a decrease in both the reliability and the resilience. In addition, the penalty value increased with an increase in the demand value, as shown in Figures 10 and 11. For instance, in scenario number one, when demand increased by 5%, the reliability and resilience decreased by approximately 7% and 3%, respectively, and the penalty value increased by 2.6%. The number of nonsupplied water days in scenario one was 51 months compared with 42 months prior to the increment in the demand.

In this work, an MSDP-ANN model is developed and applied in a case study, and the optimal results are compared with those of the CSDP-RA model. Both models are solved using same objective function and constraints with the same length of inflow data. To tackle the nonlinearity problems, the nonlinear computational method ANN is introduced as a simulation system. The results of the simulations highlight the superiority of using ANN as the simulation model (instead of the RA model) and using Q_t as the state variable in the optimization model. This approach leads to better policy when the inflow forecast for the current period is perfect. We note that Model 3 has the least penalty value. In addition, Model 3 is more reliable, resilient, and less vulnerable than the other models. To compare the accuracies of the CSDP-RA and MSDP-ANN models, the state variable was fixed as Q_t. The comparison between Models 1 and 3 showed that Model 3 increased the reliability and the resilience of the system by 8.7% and 8.2%, respectively, compared with Model 1. Therefore, it can be concluded that the MSDP-ANN model performs better than the CSDP-RA model in deriving the optimal operating policy for the reservoir system.

This research was supported by a research grant to the second author from the Ministry of Higher Education for project FRGS/1/2012/TK03/UKM/02/4 and University Kebangsaan Malaysia for project DIP-2012-03. The authors thank the Puncak Niaga (M) SDN BHD for providing the data for the Sg Langat reservoir.