In this paper, a comprehensive modified stochastic dynamic programing with artificial neural network (MSDP-ANN) model is developed and applied to derive optimal operational strategies for a reservoir. Most water resource problems involve uncertainty. To show that the MSDP-ANN model addresses uncertainty in the input variable, the result of the MSDP-ANN model is compared with the performance of a detailed conventional stochastic dynamic programing with regression analysis (CSDP-RA) model. The computational time of the CSDP-ANN model is modified with concave objective functions by deriving a monotonic relationship between the reservoir storage and optimal release decision, and an algorithm is proposed to improve the computational efficiency of reservoir operation. Various indices (i.e. reliability, vulnerability, and resiliency) were calculated to assess the model performance. After comparing the performance of the CSDP-RA model with that of the MSDP-ANN model, it was observed that the MSDP-ANN model produces a more reliable and resilient model and a smaller supply deficit. Thus, it can be concluded that the MSDP-ANN model performs better than the CSDP-RA model in deriving the optimal operating policy for the reservoir.

## INTRODUCTION

Development of a decision-making model for reservoir operation that depicts field-scale conditions is a challenging task, and a variety of different modeling approaches exist for this purpose. Many researchers have published comprehensive reviews of optimization and simulation techniques related to reservoir operations and modeling (Yeh 1985; Labadie 2004; Fayaed *et al*. 2013a, b). Chandramouli & Deka (2005) and Chandramouli & Raman (2001) developed new approaches with the decision support model (DSM) based on trained neural network models that use real-time data from previous time periods to decide on operating policies. The DSM that was developed based on an artificial neural network (ANN) outperforms the regression-based approach. Chaves & Kojiri (2007a, b) introduced a new approach for system optimization and operation known as the stochastic fuzzy neural network, which can be defined as a neuro-fuzzy system that is stochastically trained (optimized) by a genetic algorithm (GA) model to represent the system operational strategy. Moreover, to address the imprecision originated by the discretization of inflow intervals (events) in calculating the transition probabilities, the researchers applied a method based on the conditional probability of a fuzzy event.

Uncertainties associated with random properties commonly exist in a reservoir operation system (e.g. precipitation, stream inflow, and water demands); moreover, these uncertainties are further compounded by ambiguity and vagueness from the relevant hydrologic information obtained (Jing & Chen 2011). Previously, a number of stochastic dynamic programing (SDP) methods were explored to reflect the dynamic complexities and random uncertainties of the reservoir operation system (He *et al*. 2010; Housh *et al.* 2013). The current work considers the ability of ANN to mimic the nonlinearity features of the reservoir system and shows that the ANN model's capabilities are suitable for hydrological processes. The ANN is a useful tool for resolving complex problems than can be addressed using other traditional models (regression analysis (RA)). In this work, the results obtained showed that the ANN provides more accurate and reliable results than other methods, especially in the simulation result obtained on the relationships between the elevation–storage and storage–surface area capacities of the reservoir. Chaves & Chang (2008) proposed a novel intelligent reservoir operation system based on an evolved ANN. The reference to evolution indicates that the parameters of the ANN model are identified using the GA evolutionary optimization technique. Accordingly, the ANN model should represent the operational strategies of reservoir operation. This system is capable of successfully and simultaneously handling various decision variables and provides reasonable and suitable decisions.

Adeloye & Munari (2006) developed an optimization model integrated with ANN and regression models. The results showed that the latter are marginally better; however, given that the regression models require the over-year capacity to be known a priori, the ANN models are more generic and should be preferred.

An important controversy in the literature of stochastic reservoir operation and stream flow modeling concerns the appropriate statistical assumptions for the inflow sequence (Philbrick & Kitanidis 2001; Ponnambalam *et al*. 2003; Liu *et al*. 2006; Chaves & Chang 2008). The appropriateness of a stochastic assumption for this inflow sequence assumption will depend on the interval between time steps.

These remarks are cogent because many authors have adopted the independent assumption. For computational reasons (compared with the independence assumption, the first-order Markov process assumption does not add more complexity and is consistent with the solution process of the SDP formulation), the first-order Markov process description of inflows has been adopted in nearly all of the SDP formulations in statistical studies that tested the Markov assumption on rivers (Mousavi *et al*. 2005; Panagoulia 2006; Mousavi *et al*. 2007; Rani & Moreira 2010; Safavi *et al*. 2013).

In this situation, a first-order Markov process is not appropriate. If the correlations among inflows beyond lag one are strong, the occurrence of current inflows can be more precisely conditioned on previous inflows. In other words, the current inflow can be more precisely modeled using not only the inflow in the previous time step but also using the inflows in the preceding time steps (higher lags). The transition probability of the current inflow should be conditioned on additional previous inflows, which is a more precise description of reality. However, this approach yields a mathematical formulation that is impractical to solve, because these previous inflows must be included in the state variable set. These difficulties stem from the structural limitations of the SDP in which the addition of each new state variable forces a significant increase in the number of system states to be evaluated. Therefore, in SDP reservoir operation modeling, the number of state variables rarely exceeds two (one storage variable and one hydrological variable) (Raman & Chandramouli 1996; Jain *et al*. 1999; Moeini *et al.* 2011; Kumar *et al*. 2013).

An innovative approach to tackling the dimensionality problem treats the inflow sequences as inflow patterns, which are represented in the SDP formulation as a single state variable. The resultant model may be referred to as an inflow pattern stochastic dynamic programing model.

However, the efficiency and successful operation of a reservoir depends largely on the reservoir simulation system. The determination of the reservoir simulation approach must therefore be given special attention. Different methods can be used to determine the simulation, and most of them make use of statistical analysis. However, the problem with most of these methods is that they cannot easily handle the phenomenon of nonlinearity processes that occur in the reservoir (Celeste & Billib 2009; Fayaed *et al*. 2011). Simulation models associated with reservoir operation are usually based on a mass balance equation, which represents the hydrological behavior of reservoir systems using inflows and other operating conditions (Wang *et al*. 2010; Wu & Chen 2013).

This study examines the integrated ANN as a reservoir simulation component and the modified stochastic dynamic programing model (MSDP) as an optimization method in determining optimal reservoir operation. This knowledge will assist in obtaining more accurate values for optimal reservoir policy, which will act as a foundation for the case study. In addition, this work also reviews the application of the MSDP model together with other simulation models and compares the effectiveness of the different results. This work provides an understanding of better performance that will be reflected in enhanced reservoir release and an effective distribution system, which are the results of a more accurate reservoir simulation.

## Sg LANGAT DAM – CASE STUDY

This study considers the Sg Langat dam as a case study. This Malaysian dam is located geographically at latitude 3° 12′ 43.07″ north of the equator and longitude 101° 53′ 39.28″ east of the prime meridian in Kuala Lumpur. The dam was built and completed in 1979 for the purpose of water supply. The Sg Langat dam has an earth embankment made of 2.5 million cubic meters (MCM) of earth, and the crest elevation is 223.72 m with a maximum height of 61 m. The primary spillway is gated, and the crest elevation is 220.96 m. The reservoir catchment area is 41 km^{2}, as shown in Figure 1. The dam controls and regulates the volume of water released into the Langat river during the dry season to ensure that adequate water is available at the Sg Langat water treatment plant, located approximately 14 miles downstream of the dam. This dam plays a highly significant role in the neighborhood of Kajang, Bangi, because it is a major supplier of drinking water to the residents. The dam's (reservoir) capacity is 37,480.0 million L. The relationships among water surface elevation, storage volume, and water surface area constitute fundamental information that characterize the reservoir and are required in modeling studies. The elevation versus area relationship is used in evaporation computations, because evaporation volumes are a function of evaporation rates and water surface area. The elevation versus storage relationship is required to determine the head pressure in hydroelectric power computations. Storage/elevation relationships are required to relate storage capacities to elevation-based operation rule curves. The results from statistical analysis of 15 years of historical data pertaining to the Sg Langat reservoir are presented in Table 1. Table 1 shows that the average annual inflow is 46.78 × 10^{6} m^{3} with a deviation of 19.77 × 10^{6} m^{3}. The average maximum inflow occurs during the month of September, and the average minimum occurs during August. The maximum and minimum inflows realized in each month are also depicted in Table 1.

Month . | Maximum (MCM) . | Minimum (MCM) . | Average (MCM) . | Std dev. (MCM) . | Skewness . | Demand (MCM) . |
---|---|---|---|---|---|---|

Jan. | 6.61 | 0.32 | 2.75 | 2.00 | 0.34 | 4.00 |

Feb. | 3.85 | 0.46 | 2.23 | 1.17 | −0.44 | 4.00 |

Mar. | 8.14 | 1.58 | 4.05 | 1.57 | 1.26 | 2.50 |

Apr. | 5.70 | 2.50 | 4.12 | 1.17 | −0.07 | 2.00 |

May | 11.24 | 0.39 | 3.73 | 2.82 | 1.37 | 3.00 |

Jun. | 8.70 | 0.13 | 3.29 | 2.15 | 1.02 | 3.00 |

Jul. | 4.97 | 1.36 | 2.75 | 1.07 | 0.42 | 3.00 |

Aug. | 5.51 | 2.54 | 3.90 | 0.98 | 0.36 | 3.00 |

Sep. | 10.08 | 1.84 | 6.21 | 2.34 | −0.39 | 2.00 |

Oct. | 6.04 | 0.23 | 3.92 | 1.55 | −1.22 | 2.50 |

Nov. | 8.72 | 3.45 | 5.67 | 1.36 | 0.57 | 1.80 |

Dec. | 6.76 | 1.88 | 4.16 | 1.59 | 0.29 | 3.00 |

Annual | 46.78 | 19.77 |

Month . | Maximum (MCM) . | Minimum (MCM) . | Average (MCM) . | Std dev. (MCM) . | Skewness . | Demand (MCM) . |
---|---|---|---|---|---|---|

Jan. | 6.61 | 0.32 | 2.75 | 2.00 | 0.34 | 4.00 |

Feb. | 3.85 | 0.46 | 2.23 | 1.17 | −0.44 | 4.00 |

Mar. | 8.14 | 1.58 | 4.05 | 1.57 | 1.26 | 2.50 |

Apr. | 5.70 | 2.50 | 4.12 | 1.17 | −0.07 | 2.00 |

May | 11.24 | 0.39 | 3.73 | 2.82 | 1.37 | 3.00 |

Jun. | 8.70 | 0.13 | 3.29 | 2.15 | 1.02 | 3.00 |

Jul. | 4.97 | 1.36 | 2.75 | 1.07 | 0.42 | 3.00 |

Aug. | 5.51 | 2.54 | 3.90 | 0.98 | 0.36 | 3.00 |

Sep. | 10.08 | 1.84 | 6.21 | 2.34 | −0.39 | 2.00 |

Oct. | 6.04 | 0.23 | 3.92 | 1.55 | −1.22 | 2.50 |

Nov. | 8.72 | 3.45 | 5.67 | 1.36 | 0.57 | 1.80 |

Dec. | 6.76 | 1.88 | 4.16 | 1.59 | 0.29 | 3.00 |

Annual | 46.78 | 19.77 |

## MODEL FORMULATIONS AND DEVELOPMENT

### Stochastic dynamic programing model

*f*of the employed SDP model is as follows:where

*f*

^{*}

_{n}(.) = objective function;

*C*(.) = immediate return function;

*S*= storage at the beginning of time period

_{t}*t*;

*Q*= inflow during month

_{t}*t*;

*R*= release during time period

_{t}*t*;

*α*= discount rate in the SDP algorithm;

*P*= transition probability matrix.

_{r}*t*is the month index,

*N*is the operating horizon in months, and

*R*(

*t*) and

*D*(

*t*) are the respective release and demand in month

*t*. The objective function in Equation (2) is subjected to mass balance, storage, evaporation, and surplus constraints. Storage and inflow are discretized into 60 and 100 classes, respectively. The first class of the inflow state variable is represented by a value of zero. The last inflow class has no upper bound and is represented by the maximum historical inflow of the corresponding month.

*S*

_{t}_{+1}= final storage in month

*t*;

*Q*= inflow during month

_{t}*t*;

*E*= evaporation loss in the reservoir during time

_{t}*t*;

*O*= surplus from the reservoir during time

_{t}*t*.

*S*

_{max}= capacity of the reservoir in millions of cubic meters.

Month . | Maximum (MCM) . | Minimum (MCM) . | Average (MCM) . | Std dev. (MCM) . |
---|---|---|---|---|

Jan. | 38.64 | 26.6 | 37.19 | 3.2 |

Feb. | 38.64 | 28.9 | 35.93 | 3.5 |

Mar. | 38.64 | 19.8 | 34.06 | 5.73 |

Apr. | 38.68 | 14.6 | 33.7 | 7.22 |

May | 38.7 | 13 | 34.17 | 7.76 |

Jun. | 38.7 | 12.3 | 34.19 | 7.69 |

Jul. | 38.7 | 12 | 33.58 | 8.13 |

Aug. | 38.8 | 13.7 | 33.1 | 7.99 |

Sep. | 38.9 | 17.4 | 33.71 | 7.84 |

Oct. | 38.9 | 18.4 | 33.28 | 7.94 |

Nov. | 38.9 | 19.8 | 35.68 | 5.53 |

Dec. | 38.9 | 23.4 | 37.08 | 4.01 |

Month . | Maximum (MCM) . | Minimum (MCM) . | Average (MCM) . | Std dev. (MCM) . |
---|---|---|---|---|

Jan. | 38.64 | 26.6 | 37.19 | 3.2 |

Feb. | 38.64 | 28.9 | 35.93 | 3.5 |

Mar. | 38.64 | 19.8 | 34.06 | 5.73 |

Apr. | 38.68 | 14.6 | 33.7 | 7.22 |

May | 38.7 | 13 | 34.17 | 7.76 |

Jun. | 38.7 | 12.3 | 34.19 | 7.69 |

Jul. | 38.7 | 12 | 33.58 | 8.13 |

Aug. | 38.8 | 13.7 | 33.1 | 7.99 |

Sep. | 38.9 | 17.4 | 33.71 | 7.84 |

Oct. | 38.9 | 18.4 | 33.28 | 7.94 |

Nov. | 38.9 | 19.8 | 35.68 | 5.53 |

Dec. | 38.9 | 23.4 | 37.08 | 4.01 |

*t*is given as the product of the evaporation rate and the average water spread area at the beginning and end of the time period (Vedula & Mujumdar 2005). It is found that this simple equation is well suited for this particular case study. Therefore, the evaporation loss during any time period

*t*is given bywhere

*E*= evaporation loss during time period

_{t}*t*, which is computed as the function of water spread area corresponding to the active initial and final storage and evaporation rate;

*e*= evaporation rate in millimeters per month during time period

_{t}*t*;

*A*= water surface area at the top of the minimum storage level (m

_{d}^{2});

*m*= slope of the straight line of water surface area plotted against reservoir active storage;

*SA*= active initial storage at the beginning of time period

_{t}*t*;

*SA*

_{t}_{+1}= active final storage at the end of time period

*t*.

### Regression analysis

The hypothesis of regression and correlation is a classical framework used to describe relationships between two or more variables. Of course, the relationship may not necessarily be physical in nature and might also involve methods for modeling and analyzing several variables if the focus is on the relationship between a dependent variable and one or more independent variables. This analysis further explains how the typical value of the dependent variable changes when any one of the independent variables is varied and the other independent variables are held fixed. Determination of the relationships among elevation, storage, and surface area is one example of the application of RA in the science of hydrology. In investigating the relationship between two variables *x* and *y*, the realizations of the random pairs (*x _{i}*,

*y*) are plotted as a scatter plot, which illustrates the relationship between the two variables.

_{i}### Artificial neural network

*i*nodes, the hidden layer contains

*j*nodes, and the output layer contains

*k*nodes. Therefore, the output

*z*can be expressed as follows:where function

_{k}*f*is the transfer function or activation function, is the input quantity, and are the weighted values, and and are the deviations. In Equation (7), function

*f*is a type of mapping rule used to convert a neuron from a weighted input to an output and is also a type of design used to introduce nonlinear influences into the ANN network. This study chooses the most general binary logistic sigmoid function defined as follows:where the range of the value is (0, 1). If a linear function is selected for this transfer function, i.e.

*f*(

*x*) =

*x*, the entire ANN structure will become a linear influence from the input layer to output layer. The structure of the storage/surface area model is shown in Figure 2. The storage/surface area model contains 10 neurons in the input layer and five neurons in the hidden layer. Because the ANN is a supervisory learning algorithm, the optimum parameter is obtained by adjusting the weighted values in the network, where optimum means that the squared deviations between the network outputs

*zk*and the actual values or target values

*tk*achieve a minimum, i.e. the set of all available data is separated into two disjoint sets, the training set and testing set. The test set is not involved in the learning or training phase of the networks and is used to evaluate the performances of the models.

The set of all available data is separated in two disjoint sets: the training set and test set. The test set is not involved in the learning or training phase of the networks and is used to evaluate the performances of the models. In this study, 100 inputs of reservoir elevations (first 50 and last 50 inputs) were selected for calibration of the ANN model, whereas the remaining 19 inputs (in between) were used to verify or test the performance of the model. In this particular study, the structure of the ANN model is designed based on Figure 2. The structure of the elevation-area model is the same as that of the elevation-storage model, i.e. both models have the same structures. For additional details on the structure of the ANN simulation model for this reservoir, see Fayaed *et al.* (2013b).

## MODIFIED STOCHASTIC DYNAMIC PROGRAMING

During integration of the conventional stochastic dynamic programing with artificial neural network (CSDP-ANN) model, the results of the simulation and optimization model were more reliable and robust than those of the CSDP-RA, but the disadvantage of the first model is that it is computationally time consuming, which leads us to look for a modification for the CSDP-ANN model to tackle this problem. The MSDP-ANN procedure is shown and presented in Figure 3.

### Improved dynamic programing using the monotonic relationship

In conventional dynamic programing (DP), the optimal storage carry-over corresponding to a current storage *s _{t}* is determined via an exhaustive search approach, and to find for each

*s*, every

_{t}*s*

_{t}_{+1}value should be searched (Figure 4, left), i.e. if

*s*is discretized into

_{t}*n*intervals and

_{t}*s*

_{t}_{+1}into

*n*

_{t}_{+1}intervals (equal interval length), as shown in Figure 4, then to determine the optimal for

*p*, the

*n*

_{t}_{+1}discretized

*s*

_{t}_{+1}intervals should all be tested.

If the single-period utility function is concave, according to the monotonic relationship between *s _{t}* and , if we know the optimal state

*q*(at stage

*t*+ 1) corresponding to

*p*(at stage

*t*), then to search for the optimal for

*p*+ 1, we need only to test two

*s*

_{t}_{+1}values at the states of

*q*and

*q*+ 1 (Figure 4, right). Thus, by applying the monotonic relationship, the computation for determining for

*s*can be potentially reduced from

_{t}*n*

_{t}_{+1}to 2. An improved DP algorithm is proposed (as shown in Figure 5), which includes the following steps. (1) Start: discretize

*s*and

_{t}*s*

_{t}_{+1}into intervals of equal length and number them in ascending order, i.e. 1 to

*n*and 1 to

_{t}*n*

_{t}_{+1}, respectively. (2) Initialization: setting

*p*= 1 (the minimum discretized

*s*), search from 1 to

_{t}*n*

_{t}_{+1}to find its corresponding optimal , and initialize

*q*=

*s*

_{t}_{+1}*. (3) Computation: setting

*p*=

*p*+ 1, search between

*q*and

*q*+ 1, and use the better of the two to update corresponding to

*p*. (4) Repeat step 3 until

*p*=

*n*. Steps 1–4 are designed for DP computation at stage

_{t}*t*.

Using recursive computation from *T* to 1, the *T*-stage reservoir optimization operation problem can be solved using Equations (1) and (2). As illustrated in Figure 4, when *n* = *n _{t}* =

*n*

_{t}_{+1}, the computational complexity of the improved algorithm is

*n*+ 2*(

*n*–1) = 3*

*n*–2, whereas that of the conventional algorithm is

*n*

^{2}, i.e. each of the

*s*

_{t}_{+1}states (

*n*) should be tested with each of the

*s*states (

_{t}*n*). Thus, for a large number of state discretizations (

*n*), 3*

*n*–2 < <

*n*

^{2}, and the computational order of the improved algorithm can be considerably reduced.

### Monotonicity with stochastic parameters and improved SDP

*P*(

*Q*

_{t}_{+1}/

*Q*) of the reservoir inflow, which represents the conditional probability of

_{t}*Q*

_{t}_{+1}in period

*t*+ 1 on

*Q*in period

_{t}*t*(Faber & Stedinger 2001; Zhao

*et al*. 2011). Incorporating

*P*(

_{r}*Q*

_{t}_{+1}/

*Q*) into Equation (1), we obtain the recursive function of SDP as follows:

_{t}If assuming a partially concave dependence relationship between and *S _{t}*

_{+1}(i.e. ), given that the state transition probability

*P*(

*Q*

_{t}_{+1}/

*Q*) is positive, the partial dependence relationship between

_{t}*FF*

_{t}_{+1}(

*S*

_{t}_{+1}) (the weighted sum of , Equation (10)) and

*s*

_{t}_{+1}is concave (i.e. ). Thus, with a fixed

*Q*(Equation (11)),

_{t}*F*and

_{t}*FF*

_{t}_{+1}(

*S*

_{t}_{+1})│

*Q*are concave functions, and an assumption of partially concave dependence relationship between and

_{t}*s*

_{t}_{+1}leads to the same relationship between and

*S*. Hence, if assuming that

_{t}*F*(

_{t}*r*) is concave, following the procedures described in the ‘monotonicity in reservoir operation analysis’ section, we can conclude that: (1)

_{t}*F*

_{T}_{–1}(

*S*

_{T}_{–1},

*Q*

_{T}_{–1}), … ,

*F*

_{1}(

*S*

_{1},

*Q*

_{1}) are partially concave functions of

*S*

_{T}_{–1}, … ,

*S*

_{1}, respectively; and (2) with a fixed

*Q*, a monotonic relationship exists between

_{t}*S*and (and also ). If applying the monotonic relationship to improving the SDP, the procedures are shown in Figure 6 with two steps. Step 1: for a fixed value of

_{t}*Q*, run the modified DP (Figures 4 and 5) to search for corresponding to each of the possible values of

_{t}*S*. Step 2: update

_{t}*Q*and repeat step 1 until all possible

_{t}*Q*values are tested and corresponding to all (

_{t}*S*,

_{t}*Q*) combinations is identified. Using the modified SDP in the backward recursive formulation, the multistage stochastic reservoir operation problem can be solved.

_{t}### Integration of neural network stochastic dynamic programing

In this work, integration between ANN and CSDP is applied to derive the reservoir operation policy for the Sg Langat dam. The nonlinearity of natural physical processes causes a major problem in determining the simulation of reservoir systems, especially if using the conventional linear methods, e.g. RA method. To tackle this problem, a nonlinear computational method, i.e. the ANN, is introduced as a simulation system.

The model implementation procedure is applied first, and the operating storage volume (level) and stream inflows are discretized. Next, the SDP solution approach begins by initializing the values of the objective function at the last stage (a period in the future) to zero for each combination of state variables. The reservoir system is simulated using the ANN. Afterwards, the process continues backwards along the temporal stages. Each iteration consists of *T* stages that complete one annual cycle. The algorithm finds the end-of-period storage levels for each combination of the discretized beginning-of-period storage level and the average inflow of the current period. The behavior of the policy convergence after several iterations is due to the characteristics of the Markov transition probability matrix incorporated into the recursive equation.

### Model verifications

#### Failure

*t*can be in either a satisfactory (i.e. nonfailure) state NF or an unsatisfactory (i.e. failure) state F. In this study, the focus is on water supply systems, and therefore, the NF state occurs when the water supply is able to meet the water demand; hence, the F state occurs when supply cannot meet demand. Moving from time step

*t*to

*t*+ 1, the system can either remain in the same state or migrate to the other state. The duration of the

*j*th excursion into a failure period is denoted as

*d*(

*j*), and the corresponding deficit volume is denoted as

*v*(

*j*),

*j*= 1, … ,

*M*, where

*M*is the total number of failure events. The deficit volume of the failure event is calculated as the cumulative difference between demand and availability as follows:where

*d*(

*j*) is the duration of the failure, and and are the water demand and the water actually supplied, respectively. The following sections describe how to estimate R–R–V from the extracted series of failure duration and deficit volume.

It should be noted that the reliability, resilience, and vulnerability value ranges between 0 and 1. For the reliability index, a value is close to one means that the model is effective, whereas the model is effective in the case in which the value of resilience and vulnerability is close to zero.

#### Reliability

*et al*. (1982) and Park

*et al*. (2009) as follows:where

*S*is the system state variable under consideration. The most widely accepted and applied definition is occurrence reliability, which can be estimated as follows:where

*d*(

*j*) is the duration of the

*j*th failure event,

*M*is the number of failure events, and

*T*is the total number of time intervals.

#### Resilience

*et al*. (1982) defines resilience as a conditional probability as follows:where

*S*(

*t*) is the system state variable under consideration. This definition of resilience is equal to the inverse of the mean value of the time that the system spends in an unsatisfactory state, i.e.,where again is the duration of the

*j*th failure event, and

*M*is the total number of failure events. Moy

*et al*. (1986) defined resilience as the maximum consecutive amount of time that the system spends in an unsatisfactory state. To make this definition comparable with the definition of in Equation (16), resilience is expressed as the inverse of the maximum duration as follows:

#### Vulnerability

*et al.*(1982) as follows:where

*h*(

*j*) is the most severe outcome of the

*j*th sojourn into an unsatisfactory state, and

*e*(

*j*) is the probability that

*h*(

*j*) is the most severe outcome of a sojourn into the unsatisfactory state. Hashimoto

*et al*. (1982) based their vulnerability measure on the total water deficit experienced during the entire

*j*th sojourn into F, i.e. deficit volume. This definition is suited for reservoirs because the most severe outcome of a reservoir state is often the empty state,

*h*(

*j*) = 0. As a further simplification of Equation (18), both studies considered the probability of each event being equal, i.e.

*e*(1) = …=

*e*(

*M*)

*=*1

*/M*, where

*M*is the number of failure events; therefore, they estimated vulnerability as the mean value of the deficit events

*v*(

*j*) as follows:

#### Cumulative penalty

*n*is the number of years of simulation (15 in this study),

*R*is the actual release during the time period

_{ji}*i*of the year

*j*(and equals the optimal release that was suggested by the model whenever possible), and

*D*is the demand for the time period

_{i}*i*.

## RESULTS AND DISCUSSION

This section presents the results of applying the models to the case study. A 15-year time series of stream flow (1996–2010) is used for evaluation of the model for water allocation from the Sg Langat reservoir according to the water treatment plant demands. In all models, the performance indices are used to evaluate the results of the models. These indices show how often the system does not fail (reliability), how quickly the system returns to a satisfactory state once a failure has occurred (resiliency), and the significance of the consequences of failure (vulnerability). With respect to the problem statement, two different types of state variables (*Q _{t}* or

*Q*

_{t}_{–1}) were used when each model was tested (CSDP-RA and CSDP-ANN). From these tests, four submodels were obtained and are given in Table 3.

Model . | Description . | State variable . |
---|---|---|

1 | CSDP-RA | Q _{t} |

2 | CSDP-RA | Q_{t}_{−1} |

3 | MSDP-ANN | Q _{t} |

4 | MSDP-ANN | Q_{t}_{−1} |

Model . | Description . | State variable . |
---|---|---|

1 | CSDP-RA | Q _{t} |

2 | CSDP-RA | Q_{t}_{−1} |

3 | MSDP-ANN | Q _{t} |

4 | MSDP-ANN | Q_{t}_{−1} |

### Conventional stochastic dynamic programing with the regression analysis model

However, the results from these models were not as accurate compared with the results of other methods used in this study.

To generate the operating policies for reservoirs, a CSDP-based model with the RA was developed. The results of the CSDP-RA model with *Q _{t}* and

*Q*

_{t}_{–1}as the state variables are presented in Table 4. In this table, the reliability, resiliency, and vulnerability (used as the performance indices for supply to the water demand at the downstream section of the reservoir) for a 15-year simulation period were 0.699%, 0.302%, and 0.263 MCM, respectively, when

*Q*was used as state variable (Model 1). However, when the state variable was

_{t}*Q*

_{t}_{–1}(Model 2), the performance indices were 0.678%, 0.289%, and 0.277 MCM, respectively, when equal numbers of inflow and reservoir storage discretization were considered. The results show that in using Model 1, the reliability of supplying the demands improved by 3.1%. The implication for this result is that out of the 175 months in the study period, the total number of nonsupplied demand months (unsatisfactory states) is 52 months using Model 1 but only 56 months using Model 2. In addition, more robust and reliable results were obtained. As given in Table 4, the resilience in Model 1 can produce a gain of 4.4% compared with that of Model 2. The vulnerability values, i.e. the expected values of nonsupplied water demand, were approximately 0.263 and 0.277 MCM for Models 1 and 2, respectively. The penalty function was used as a measure of the performance of the reservoir system and also used to identify the decision that makes the greatest contribution to the operation of the reservoir. The objective is to release the amount of water that is closest to the target demand, and thus, the operational performance also can be measured in terms of the deviation of the release decision from its target demand. The penalty function attempts to minimize the functional value and forces the release to satisfy the target demand. From Table 4, the penalty values for Models 1 and 2 are 1.59 × 10

^{–5}and 1.67 × 10

^{–5}, respectively. This result shows that the penalty value increased by 5% for Model 1 compared with that of Model 2.

Model . | Reliability (%) . | Resilience (%) . | Vulnerability (MCM) . | Penalty . |
---|---|---|---|---|

1 | 0.699 | 0.302 | 0.263 | 1.59 × 10^{–5} |

2 | 0.678 | 0.289 | 0.277 | 1.67 × 10^{–5} |

Model . | Reliability (%) . | Resilience (%) . | Vulnerability (MCM) . | Penalty . |
---|---|---|---|---|

1 | 0.699 | 0.302 | 0.263 | 1.59 × 10^{–5} |

2 | 0.678 | 0.289 | 0.277 | 1.67 × 10^{–5} |

### Conventional stochastic dynamic programing with the artificial neural network model

In this model, integration between the ANN and CSDP was used to derive the reservoir operation policy in Sg Langat. To evaluate the performance of the CSDP-ANN model, the results of the performance indices are determined and compared. The reliability, resiliency, and vulnerability of supplying the water demand at the downstream section of the reservoir are 0.76%, 0.327%, and 0.235 MCM, respectively, for Model 3, whereas the values for the same parameters are 0.734%, 0.310%, and 0.244 MCM, respectively, for Model 4, considering the same numbers of inflow and reservoir storage discretization. The results of the performance indices in supplying the demand to each sector are presented in Table 5. The results show that Model 3 improved the reliability of supply by 3.5%, which means that the total number of nonsupplied demand months (unsatisfactory states) is 47 months when *Q _{t}*

_{–1}was the state variable (Model 4) and approximately 42 months when

*Q*was the state variable (Model 3). The resilience in Model 3 shows a gain of 5.4% compared with that of Model 4. The vulnerability values are approximately 0.235 and 0.244 MCM, respectively, in Models 3 and 4. The penalty value in Model 3 is much better than that of Model 4, and its gain was 4.5%; this result demonstrates that Model 3 is more reliable and robust in deriving the optimal operation policy for the Sg Langat reservoir. The number of storage is 60, and from Table 6, the computation time at this storage using the CSDP-ANN model was 365.5 minutes while it was 18 minutes when the MSDP-ANN model was used.

_{t}Model . | Reliability (%) . | Resilience (%) . | Vulnerability (MCM) . | Penalty . |
---|---|---|---|---|

3 | 0.760 | 0.327 | 0.235 | 6.08 × 10^{–6} |

4 | 0.734 | 0.310 | 0.244 | 8.80 × 10^{–6} |

Model . | Reliability (%) . | Resilience (%) . | Vulnerability (MCM) . | Penalty . |
---|---|---|---|---|

3 | 0.760 | 0.327 | 0.235 | 6.08 × 10^{–6} |

4 | 0.734 | 0.310 | 0.244 | 8.80 × 10^{–6} |

No. of storage . | CSDP-ANN . | MSDP-ANN . |
---|---|---|

10 | 9.5 | 2.8 |

20 | 38.4 | 5.7 |

40 | 152.2 | 11.7 |

60 | 365.5 | 18.0 |

80 | 648.3 | 23.6 |

100 | 982.2 | 29.5 |

No. of storage . | CSDP-ANN . | MSDP-ANN . |
---|---|---|

10 | 9.5 | 2.8 |

20 | 38.4 | 5.7 |

40 | 152.2 | 11.7 |

60 | 365.5 | 18.0 |

80 | 648.3 | 23.6 |

100 | 982.2 | 29.5 |

### Comparisons between CSDP-RA and MSDP-ANN models

The CSDP-RA and MSDP-ANN were applied to generate reservoir operation policies. The results from the two models are compared in this section, as shown in Figures 7 and 8. The cumulative penalty values for each of the four models are presented in Figure 7. It can be observed that Model 3 has the lowest penalty value. In addition, from the data shown in Figure 8, Model 3 is more reliable, more resilient, and less vulnerable than the other models. To compare the accuracies of CSDP-RA and MSDP-ANN models, the state variable *Q _{t}* was fixed. Comparison between Models 1 and 3 showed that the latter increased the reliability and the resilience of the system by 8.7% and 8.2%, respectively, which means that the total number of nonsupplied demand months (unsatisfactory states) is 53 months in Model 1 and approximately 42 months in Model 3. The nonlinearity of natural physical processes constitutes a major barrier to successful simulation of reservoir systems, especially if using such conventional linear methods as the RA method. To surmount this difficulty, the ANN nonlinear computational method is introduced for simulation of the system. The results of this comparison highlight the superiority of using ANN as the simulation model and

*Q*as the state variable in the optimization model. The result of applying this reservoir operation system to each month is a three-pronged relationship among storage number, monthly inflow, and final storage number. This relationship represents the steady-state policy for each prospective month. In Figure 9, February was taken as an example to illustrate the shape of the relationship.

_{t}### Applying different demand scenarios to Model 3

From the above discussions, it can be concluded that the best model for deriving the optimal operation policy is MSDP-ANN with *Q _{t}* as the state variable (Model 3). In this study, experiments were conducted to investigate the behavior of the system under different demand scenarios with increments of 5, 10, 15, 20, and 25% in the demand target, as given in Table 7. The results showed that an increase in the demand led to an increase in the vulnerability and a decrease in both the reliability and the resilience. In addition, the penalty value increased with an increase in the demand value, as shown in Figures 10 and 11. For instance, in scenario number one, when demand increased by 5%, the reliability and resilience decreased by approximately 7% and 3%, respectively, and the penalty value increased by 2.6%. The number of nonsupplied water days in scenario one was 51 months compared with 42 months prior to the increment in the demand.

Model 3 (%) . | Reliability . | Resilience . | Vulnerability . | Penalty . |
---|---|---|---|---|

5 | 0.709 | 0.316 | 0.246 | 6.24 × 10^{–6} |

10 | 0.663 | 0.309 | 0.249 | 7.51 × 10^{–6} |

15 | 0.633 | 0.305 | 0.253 | 8.29 × 10^{–6} |

20 | 0.617 | 0.300 | 0.267 | 8.96 × 10^{–6} |

25 | 0.596 | 0. 294 | 0.278 | 9.83 × 10^{–6} |

Model 3 (%) . | Reliability . | Resilience . | Vulnerability . | Penalty . |
---|---|---|---|---|

5 | 0.709 | 0.316 | 0.246 | 6.24 × 10^{–6} |

10 | 0.663 | 0.309 | 0.249 | 7.51 × 10^{–6} |

15 | 0.633 | 0.305 | 0.253 | 8.29 × 10^{–6} |

20 | 0.617 | 0.300 | 0.267 | 8.96 × 10^{–6} |

25 | 0.596 | 0. 294 | 0.278 | 9.83 × 10^{–6} |

## CONCLUSIONS

In this work, an MSDP-ANN model is developed and applied in a case study, and the optimal results are compared with those of the CSDP-RA model. Both models are solved using same objective function and constraints with the same length of inflow data. To tackle the nonlinearity problems, the nonlinear computational method ANN is introduced as a simulation system. The results of the simulations highlight the superiority of using ANN as the simulation model (instead of the RA model) and using *Q _{t}* as the state variable in the optimization model. This approach leads to better policy when the inflow forecast for the current period is perfect. We note that Model 3 has the least penalty value. In addition, Model 3 is more reliable, resilient, and less vulnerable than the other models. To compare the accuracies of the CSDP-RA and MSDP-ANN models, the state variable was fixed as

*Q*. The comparison between Models 1 and 3 showed that Model 3 increased the reliability and the resilience of the system by 8.7% and 8.2%, respectively, compared with Model 1. Therefore, it can be concluded that the MSDP-ANN model performs better than the CSDP-RA model in deriving the optimal operating policy for the reservoir system.

_{t}## ACKNOWLEDGEMENTS

This research was supported by a research grant to the second author from the Ministry of Higher Education for project FRGS/1/2012/TK03/UKM/02/4 and University Kebangsaan Malaysia for project DIP-2012-03. The authors thank the Puncak Niaga (M) SDN BHD for providing the data for the Sg Langat reservoir.