Using long short-term memory networks for river flow prediction

The rainfall–runoff process of a river basin is normally characterized by a high degree of nonlinearity. The process is one of the most important components in the hydrological cycle and its accurate modelling is crucial for water resources and flood management (Clarke 1994; Nourani 2017). Rainfall–runoff models are usually classified into three main classes: distributed, conceptual and black box models (Clarke 1994). Distributed and conceptual models are based on various hydrological processes; however, they are limited by our understanding and ability to accurately represent these processes and computational resources. By contrast, black box models are normally data-driven but can provide an accurate prediction in many situations (Tanty & Desmukh 2015; Nourani 2017). Artificial neural network (ANN) models are one of the typical black box models. Since Daniell (1991) applied ANNs to streamflow modelling, ANNs have been widely applied in hydrological modelling because of its strong non-linear fitting ability (ASCE Task Committee 2000a, 2000b). Currently, various ANN models have been employed to study the rainfall–runoff process, such as fuzzy neural networks (Nayak et al. 2004), wavelet neural networks (Wang & Ding 2003; Alexander & Thampi 2018) and Bayesian neural networks (Bateni et al. 2007; Kayabası et al. 2017). Traditionally, the ANN learns the relationships between input and output variables from historical data provided and does not have the ability to automatically select the input variables or factors. ANNs with multiple hidden layers have excellent learning ability (Hinton & Salakhutdinov 2006), thus, deep neural networks are increasingly used in hydrology to simulate the rainfall–runoff relationships building on big data which have become available in recent years (Hu et al. 2018, 2019; Kratzert et al. 2019; Le et al. 2019).

In recent years, deep learning has made significant advances in the field of machine learning and data science (Negnevitsky & Pavlovsky 2005). Many deep learning algorithms have shown great potential in solving real-world problems (Khan & Yairi 2018), for example, recurrent neural networks (RNN) (Shin et al. 2017) and convolutional neural networks (Zhou et al. 2019). In particular, the RNN network has a strong learning ability for time series data (Bengio et al. 1994; Hochreiter & Schmidhuber 1997; Saon & Picheny 2017) as it includes loops to allow the information from previous time steps to be passed to the next time step. However, the gradient disappearance or explosion problem makes the RNN gradually lose the ability to learn long-distance information (Bengio et al. 1994). To overcome the deficiency, the long short-term memory (LSTM) network, a special type of RNN, was developed for learning with long sequence data (Hochreiter & Schmidhuber 1997), as it is capable of learning long-term dependencies in the data series. Based on the concept of LSTM networks, many similar networks have been constructed to improve the learning ability for different tasks (Sutskever et al. 2013; Bellegarda & Monz 2016). At present, the LSTM has been successfully used in speech recognition and text translation (Bellegarda & Monz 2016; Rocha et al. 2019).

In the last few years, LSTM networks have been tested and studied in watershed hydrological modelling, and their potential has been demonstrated in many applications, such as river flow and flood predictions (Shen 2018). Kratzert et al. (2018) applied the LSTM network to simulate the daily flows of 241 basins and found that it greatly outperforms hydrological models that are calibrated both at the regional level and at the individual basin level. Lee et al. (2018) developed an LSTM for daily runoff simulations based on the water level data of 10 stations at the upper Mekong River and showed that the LSTM performs better than the Soil and Water Assessment Tool model (SWAT). Zhang et al. (2018) compared four different neural networks, namely multilayer perceptron, wavelet neural network (WNN), LSTM and gated recurrent unit (GRU), in predicting the daily discharges of combined sewer overflow structures, and showed that LSTM and GRU are highly capable of multi-step-ahead time series prediction. Sahoo et al. (2019) applied LSTM to forecast daily flows during low-flow periods in the Mahanadi River basin, India. Kratzert et al. (2019) proposed an Entity-Aware-LSTM (EA-LSTM), which performed substantially better at the regional level with 531 basins than several hydrological models calibrated individually for each basin. Hu et al. (2018) tested an LSTM model on 98 flood events and indicated that the LSTM model outperformed conceptual and physical models. Yan et al. (2019) constructed an LSTM with historical flow and weather data and weather forecasts and indicated that the LSTM outperforms support vector machines in flood predictions, especially for flood peak flow forecasts. Karimi et al. (2019) compared three data-driven methods, i.e., ANN, LSTM and Least Absolute Shrinkage and Selection Operator (LASSO) for flood flow predictions in sewer systems, and concluded that all three models provide acceptable prediction performance, but LSTM outperforms ANN due to the inherent memory integrated with a feedback structure. Muhammad et al. (2019) proposed a hybrid model by combining LSTM and GRU for river flow simulations, which was used for early flood warning. Hu et al. (2019) integrated an LSTM and reduced order model to represent the spatial–temporal distribution of floods. Le et al. (2019) used the LSTM in modelling 1-, 2- and 3-day flood events in Vietnam's Da River basin. In summary, previous research has shown the ability of LSTM in river flow predictions. However, there is a lack of understanding on how LSTM structures and parameters are linked to predictive accuracy in hydrological modelling.

The main aim of this paper is to assess the performances of LSTM networks in river flow predictions in terms of LSTM structures and parameters. In this study, the LSTM networks with different network structures, i.e., fully connected layers and LSTM cells are trained and their performances compared using two case studies of different characteristics – the Hun river basin and the upper river basin of Yangtze River, China. The trained LSTM networks are used to predict the river flows in the two case study river basins. Finally, the LSTM networks are compared with four models, i.e., the SWAT, Xinanjiang model (XAJ), multiple linear regression model (MLR) and back-propagation neural networks (BP).

In this section, the LSTM network for flow simulation and predication is first presented and the key components including the network structure, LSTM cells and loss function are explained. Then the data pre-processing and evaluation criteria used in this study are introduced. Finally, different simulation scenarios designed to study the performance of the LSTM network are explained.

In the LSTM network, the key components are fully connected layers and LSTM cells. As shown in Figure 1, the LSTM network contains four types of layers: (a) the input layer which receives the input sequence data; (b) the fully connected layer a which transfers the dimensions of the input data into the dimensions of LSTM cells and establishes a bridge between the input layer and the LSTM cell layer; (c) the LSTM cell layer of n cells (i.e., simple networks) which provides different memory abilities; (d) output layers including fully connected layers b1, b2 and the output flow vector, which transfer the outputs of LSTM cells to flows.

Figure 2 shows the structure of the LSTM cell. There are two key states in LSTM cell calculation, i.e., cell state and hidden state. In Figure 2, C_t−1 and H_t−1 represent the cell state and hidden state at time step t− 1, respectively. The cell state is the main chain of the data stream, which allows the data to flow forward substantially unchanged. However, the data in the hidden state can be added or removed from the cell state (Le et al. 2019), which is carefully controlled by ‘forget gate’, ‘input gate’ and ‘output gate’, represented by the dashed boxes in Figure 2. The gates are neural network layers with a series of matrix operations, which contain different individual weights and biases. The LSTM cell uses gates to control the memory process to avoid the long-term dependency problem (Hochreiter & Schmidhuber 1997). The cell learns from time series data using five simple network layers, including three sigmoid layers and two tanh network layers (Le et al. 2019).

The pseudo code of the LSTM network training is shown in Algorithm 1. In the pseudo code, the parameters include the batch size of training input data (T), the dimensions of the input data (m), the dimensions of the output flow vector (g), LSTM cell size (n) and the input and output dimensions of the fully connected layers (i.e., h and k). In the case study, the LSTM network training ends after 1,500 epochs.

In this section, the scenarios of different LSTM structures and parameters are presented to test the network performances as shown in Table 1. The LSTM network structure in Figure 1 is taken as scenario A1 and used as a benchmark for comparison of the other network structures. Building on scenario A1, Scenario A2 has a fully connected neural layer with an activation function added between the input layer and the fully connected layer a. Scenario A3 is developed by removing the activation functions in the fully connected layers b1 and b2 based on A1.

Comparing the performances between scenario A1 and A3 allows us to analyse the impact of activation functions which establish a non-linear transformation between the LSTM cells and the output runoff vector. To evaluate the impacts of the number of layers on the learning efficiency of LSTM, scenarios B1 and B2 are constructed by adding one and two fully connected layers to the benchmark A1, respectively.

The batch sizes of training samples (T) and number of LSTM cell (n) are important network parameters, which determine the learning efficiency of the LSTM network. Table 1 shows their values tested in this study.

In this study, two basins are taken as case studies, the Hun river basin and the upper Yangtze river basin, which are located in the northeast and southwest of China, respectively (Figure 3). The basic characteristics of the two basins are shown in Table 2.

The Hun river originates from the Changbai Mountain. The precipitation is affected by temperate monsoon continental climate. The vegetation in this basin is well maintained and is rarely interrupted by human activities. 70% of the annual precipitation occurs from June to September.

The Upper Yangtze river originates from the Qinghai-Tibet Plateau. The drainage area above the Three Gorges dam in the Yangtze River is the upper reach. The precipitation of the basin is affected by the topography and the monsoon, which makes the annual precipitation unevenly distributed both spatially and temporally. The precipitation increases from northwest to southeast, and about 70% of the precipitation occurs from May to September.

In the Hun river, the 10-day average meteorological data from 10 stations (i.e., m = 10) and river flow data from the Huanren station (HR) (i.e., g = 1) from 1970 to 2010 were obtained. The 30-year data from 1970 to 1999 are used to train LSTM networks and calibrate the comparison models which are introduced in Section Comparison models. The data from 2000 to 2010 are used to verify the performances of the models.

In the Upper Yangtze river, the daily meteorological data were obtained from 1991 to 1998, including 57 meteorological stations (i.e., m = 57). The observed daily streamflow from six hydrological stations (i.e., g = 6) as shown in Figure 3 was obtained from 1991 to 1998, i.e., Shi gu (SG), Pan zhi hua (PZH), Bei bei (BB), Wu long (WL), Xiang jia ba (XJB) and Wang zhou (WZ). The data from 1991 to 1995 are used for model training and calibration, and the data from 1996 to 1998 are used for verification.

In the Hun river basin case study, four models are constructed as comparison models to evaluate the performance of the proposed LSTM network, i.e., SWAT, XAJ, MLR and BP. In the upper Yangtze river basin, SWAT is used only.

The SWAT model is a distributed hydrological model developed by the U.S. Department of Agriculture and the Agricultural Research Service (Guzman et al. 2015). The SWAT model has strong physical mechanisms. In this study, the DEM raster data with a resolution of 90 m and the land use raster data with a resolution of 3 km were obtained in Hun river basin. The DEM and land use data of 3 km × 3 km in the Upper Yangtze river basin were obtained. In this study, the SWAT model is calibrated using the Calibration Uncertainty Procedure (SWAT-CUP) program, which is an auto-calibration tool that allows for sensitivity analysis, calibration, validation and uncertainty analysis of the SWAT model (Abbaspour 2011; Abbaspour et al. 2015). Finally, the key parameters for the two basins are listed in Table 3.

The XAJ model is a conceptual rainfall–runoff model, which was developed by Zhao (1992). This model has been widely used in China, particularly in humid and semi-humid regions (Xu et al. 2014). The XAJ model assumes that runoff is not produced until the soil water content of the aeration zone reaches its field capacity. The actual evapotranspiration is computed from potential evapotranspiration while soil storage deficit is calculated in three layers, i.e., upper, lower and deep soil layers. The XAJ model parameters of the Hun river basin have been calibrated using a Genetic Algorithm (Xu & Peng. 2016). In calibration, NSE andRE are used to evaluate the performance of the parameters as shown in Equations (14) and (15), respectively.

The calibrated parameters of the XAJ model are shown in Table 4. In this model, the surface runoff, interflow and groundwater are routed using instantaneous unit lines, and the parameters of the three lines, i.e., (n and k) are set to (3, 4), (4, 5) and (4.5, 7), respectively.

In the MLR model, one regression model was developed for each time step to identify the factors of physical and statistical significance in the Hun river basin. There are 36 time steps in a year, i.e., 10 days for each time step. In the MLR models, the least square method is used to estimate the parameters of the regression equations based on the observed runoffs and factors, and the equations are shown in Table 5.

The BP model takes the flows at time step t− 2 and t− 1 and the average precipitation of the watershed at time step t− 2, t− 1 and t as inputs and takes the flow at time step t as output. The BP model constructed using four-layer neural network, and the network nodes for each layer are 5 (input layer), 50 (hidden layer), 50 (hidden layer) and 1 (output layer), respectively. The outputs of the hidden layer are transformed by the sigmoid activation function. The BP network was trained for 900 epochs as it was already converged.

The network structure and parameters have a great influence on the learning efficiency. In this study, the LSTM networks are tested on the Hun and Upper Yangtze river basins, respectively. First, the effects of LSTM network structure are analysed and the performances of the network parameters are evaluated in terms of the number of cells (n) and the batch size (T). Then, the structure and parameters with the best performance are selected to predict the river flows of the two study cases. Finally, the performances of the LSTMs are compared with the results from the comparison models.

Scenarios A1, A2 and A3 are trained for the Hun river and Upper Yangtze river case studies, and the variations of the loss function values are shown in Figure 4. For each network structure, the network weights are trained for 1,500 epochs. The training loss values of each scenario in Figure 4 are the average values of 10 independent training runs. Note that the model parameter values used for the results in Figure 4 are those optimal values identified in Table 6. The results from the two case studies in Figure 4 show the following key points:

• (1)

  The loss value of A1 is rapidly decreased, demonstrating a rapid learning.

• (2)

  The inputs in A2 are nonlinearly transferred by activation functions before being passed to the LSTM cells; as a result, the LSTM cells cannot effectively capture the long-term time dependencies in the time series data. Thus, the loss values cannot be reduced rapidly during the training.

• (3)

  In A3, there is only a simple linear transformation between the LSTM cells and output layer. This makes learning difficult with a slow reduction in loss values before they start to increase after 500 epochs.

The results from Figure 4 show the LSTM structures in Scenarios A2 and A3 could not provide efficient learning and the model outputs cannot match the target values well.

The test results from Scenarios B1 and B2 are shown in Figure 5. The results indicate that all the scenarios can learn effectively in the two case studies. However, the detailed loss value variations between epochs 1,250 and 1,500 as shown in Figure 5(a2,b2) reveals that scenario A1 converges faster and is more stable than the other two scenarios. This implies that the network does not need a large number of fully connected layers between the LSTM cells and the output layer to improve the learning efficiency. Therefore, the LSTM structure in A1 is selected to predict the river flows in the two case study basins.

In the LSTM network training, the batch size (T) of training samples has a significant effect on the learning efficiency. In this study, a number of T values are tested for Hun river (i.e., 10, 30, 50, 70 and 90) and Upper Yangtze river (i.e., 30, 60, 120, 180 and 360). Figure 6 shows the training loss variations of the Hun river and Upper Yangtze river. The loss values fluctuate dramatically when the batch size is small, i.e., T = 10 or 30 in the case of Hun river and T = 30, 60 or 120 in the case of Upper Yangtze river. This indicates that the LSTM cells cannot capture the periodicity in the time series using small batch sizes, when the network weights are updated frequently. With T increasing, the amount of training samples used for learning can gradually reflect the periodicity. Therefore, the fluctuation of the loss values is gradually reduced. Figure 6(a2,b2) shows the magnified training loss curves during epochs from 1,250 to 1,500. The results indicate that the learning curves are stable after T = 50 and 180 for the Hun river and Upper Yangtze river basins, respectively.

Figure 7 shows the means of the loss values from epochs from 1,250 to 1,500 in Figure 6. The results show that the loss means are gradually decreasing with T increasing, which represent the learning efficiencies of the LSTM network are gradually improved. In the case of Upper Yangtze river, when T > 180, the loss value cannot be reduced.

The LSTM cell is the core concept in the LSTM network. The number of the cells (n) determines the performances of the network. Figure 8 shows the means of the loss values from epochs 1,250 to 1,500 using different cell numbers. With the Hun river basin, when the number of cells reaches 20, LSTM achieves a good learning efficiency as shown in Figure 8(a). When the cell number exceeds 20, the efficiency shows very slow improvement. Figure 8(b) reflects the network performances in the Upper Yangtze river. When the number of cells reaches 40, the learning efficiency begins to stabilize. In this study, 20 and 50 cells are used in the LSTM networks for the Hun river and the Upper Yangtze river, respectively.

Based on the analysis of the structure and parameters of the LSTM network as above, the structure and parameters with the best performance as shown in Table 6 are selected to learn and predict the flows of the Hun river and Upper Yangtze river, respectively.

Figure 9(a) shows the observed and simulated flows from 1970 to 1999, and the performance evaluation criteria are shown in Table 7. The simulated flows in Figure 9(a) fit well with the observed flows. In training, the LSTM achieves an NSE value of 0.98. The results show that LSTM has a strong non-linear learning ability and it outperforms the other models in the Hun river basin. During the models, the MLR model performs worst.

In the verifying process, the predictive ability of the LSTM is shown in Table 7 to be slightly worse than that of the SWAT. However, the LSTM slightly outperforms the XAJ model in terms of NSE but is much better than other models. Figure 9(b) shows the predicted and observed flows from 2000 to 2010. Though the predicted flows from the LSTM did not match well with the observed in some periods, most peak flows are predicted well. This is clearly shown in the scatter plots of the observed and simulated flows from the training and verification periods in Figure 10. Though the overall performances of SWAT and XAJ in terms of the three criteria are better than those of LSTM, LSTM performs much better for peak flows, which are of particular concern in flood predictions.

In the Upper Yangtze river basin, the daily data of 57 meteorological stations are used as inputs, and the daily flow of six hydrological stations is used as target values. The flows of the six stations are simulated using LSTM and SWAT and their performances are shown in Table 8.

In the training, LSTM has a very high performance for the flows of six stations. The NSE and R² values indicate that the LSTM outperforms the SWAT during training. Figure 11 shows the simulated flows at a station (WZ).

The scatter plots of simulated and predicted flows for six hydrological stations are shown in Figure 12. The results indicate that the performance of LSTM in the verifying period is worse than that in the training period. Predicted peak flows are likely to be lower than those observed.

Note that in Upper Yangtze river, the LSTM network is constructed to predict flows at multiple stations at the same time. The training results show that the LSTM network has a strong fitting ability to learn the flow data of multiple hydrological stations.

LSTM network training is significantly affected by the training dataset size. It is generally understood that the network training requires a large amount of training sample data. However, the dataset size depends on the characteristics of the catchment and flows of concern, which determines the complexity of the input–output relationships represented by the LSTM. The LSTM has been shown to perform well with a smaller dataset than 30 years used in the Hun river basin. For example, Kratzert et al. (2018) used the daily meteorological data and observed flow data from 15 years in 241 catchments to train LSTM networks, whose performances are comparable to process-based models. Lee et al. (2018) trained a deep LSTM network using daily rainfall and water level data during two periods: 2000–2002 and 2008–2014 (a total of 9 years) and it outperformed a SWAT model when verified using daily data from 2003 to 2007.

A main disadvantage of using large datasets is the computational time required for LSTM training, particularly when individual networks are trained for a large number of catchments. This issue could be tackled from several aspects: (1) using the advances in computing power such as Graphics Processing Units and cloud computing; (2) pooling the datasets from individual catchments of similar hydrological characteristics to train an LSTM network as a regional model which can predict the discharge for a number of catchments in the region (Kratzert et al. 2018); (3) training the network off-line for real-time predictions. The second aspect is also important for flow predictions in ungauged catchments as suggested by Kratzert et al. (2018) and provides a new application area in the use of the LSTM network in hydrological predictions.

Transfer learning is powerful and useful in deep learning as it can use the network knowledge gained from solving one problem to help solve another similar problem. Due to the focus of this work, transfer learning is not investigated here. Future research should explore the potential of transfer learning from two aspects: (1) building on a reference architecture (e.g. Scenario A1 in this study), the network knowledge (e.g. parameters) could be applied to other similar architectures in solving the same problem so training could be continuously improved using prior network knowledge; (2) transferring the LSTM learning knowledge from data-rich catchments to data-scarce catchments, so the flow predictions in data-scarce catchments could be improved.

In this study, the performance of LSTM networks is assessed for river flow simulations using two river basins: the Hun river and Upper Yangtze river. Different LSTM structures are analysed. The prediction performances are compared against other models including hydrological models and data-driven models. The key research conclusions are summarized below.

In the LSTM network, the activation function in the fully connected layer before the LSTM cell layer can substantially reduce learning efficiency. In the LSTM network, the transformation through activation functions weakens the correlations between precipitation and flow, leading to failure in learning the rainfall–runoff relationships in both study basins. On the contrary, non-linear transformation following the LSTM cells is required to improve learning efficiency due to the different magnitudes of precipitation and streamflow. Further, increasing the number of fully connected layers cannot improve the learning efficiency, instead it needs more epochs due to the fluctuation in the loss values.

In the LSTM network, the batch size and the number of LSTM cells are the sensitive parameters that affect the learning efficiency. The results of this study show that the learning efficiency continues to increase with the batch size and the number of cells increasing. However, when the learning is stable, the number of cells should be kept to the minimum to reduce the complexity of the network.

The LSTM has superior non-linear learning ability for time series data and has a simple structure and few parameters, which has strong application potential in streamflow simulation. The results of this study show that the non-linear learning ability in the training process is very powerful.

The LSTM networks achieve good performance compared to other models considering three criteria, i.e., NSE, R² and RE. In the case of Hun river, LSTM outperforms MLR and BP and achieves a similar level of accuracy of XAJ. It is slightly worse than a well-calibrated SWAT but provides more accurate predictions for peak flows. In the case of the Upper Yangtze river, LSTM outperforms SWAT during the training but is worse than SWAT in the verification period. This is mainly because predicted peak flows are likely to be lower than those observed.

This research is supported by the National Natural Science Foundation of China (Grant No. 51609025 and 51709108) and the UK Royal Society through an industry fellowship to Guangtao Fu (Ref: IF160108) and an international collaboration project (Ref: IEC\NSFC\170249). Guangtao Fu is also supported by The Alan Turing Institute under the EPSRC (Grant EP/N510129/1). The Open Fund Approval (SKHL1713, 2017), Chongqing technology innovation and application demonstration project (cstc2018jscx-msybX0274, cstc2016shmszx30002). We would like to thank the Hun river cascade hydropower reservoirs development Ltd and Upper Yangtze River Bureau of Hydrological and Water Resources Survey for the case study data.

All relevant data are available from an online repository or repositories (http://www.hydroshare.org/resource/93f1f580de88403a8c52d2b3238297eb).