A hybrid deep learning approach for streamflow prediction utilizing watershed memory and process-based modeling

Reliable streamflow prediction is of paramount importance for effective water resource allocation and disaster management (Erdal & Karakurt 2013; Chu et al. 2021; Reis et al. 2021). In this regard, hydrological models, including data-driven, conceptual, and physically based, play a pivotal role in enhancing our comprehension of the watershed physical processes and how these processes link (Bourdin et al. 2012). Despite criticisms for parametric complexity and challenging implementation, process-based watershed models emerge as indispensable when a comprehensive grasp of hydrological processes and region water balance is crucial (Fatichi et al. 2016). Process-based models' complexity introduces certain drawbacks, such as heightened uncertainty, calibration and validation challenges, and modeling costs (Moges et al. 2020; Herrera et al. 2022).

Data-driven modeling has emerged as a viable alternative to traditional hydrological techniques (Fahimi et al. 2017; Lee et al. 2023a). These approaches leverage the power of data to uncover patterns from time-series input–output data, thereby reducing the reliance on extensive physical data inputs (Karandish & Šimůnek 2016). In essence, data-driven models are ‘black-box’ methodologies that establish relationships between inputs and outputs without explicitly modeling the underlying physical hydrological processes (Solomatine & Ostfeld 2008; Bourdin et al. 2012; Chu et al. 2021). Despite this limitation, data-driven models have consistently shown superior performance compared to traditional approaches (Nearing et al. 2021; Reis et al. 2021; Yu et al. 2023). This performance advantage has fueled a surge in the adoption of data-driven modeling for hydrological analysis (Yaseen et al. 2015; Zounemat-Kermani et al. 2020; Mohammadi et al. 2024).

However, in addition to their inherent black-box nature, data-driven models have faced criticism for their vulnerability to overfitting or underfitting, which is contingent upon the model's complexity, as well as the volume and quality of the data (Ghaith et al. 2020; Mudiyanselage Viraj et al. 2021). Furthermore, both data-driven and process-based modeling approaches exhibit suboptimal performance when predicting extreme events (Zheng et al. 2018; Tan et al. 2020; Zounemat-Kermani et al. 2021). Bridging the gap, expert knowledge-based approaches have surged in popularity, offering both enhanced performance and interpretability for accurate prediction (Mudiyanselage Viraj et al. 2021; Willard et al. 2023). In hydrology, these methods involve integrating watershed memory indicators such as baseflow analysis with other input data (Zemzami & Benaabidate 2016; Tongal & Booij 2018; Li et al. 2022), simulating intermediate variables (Humphrey et al. 2016; Noori & Kalin 2016), and accounting for residual errors (Tian et al. 2018; Kassem et al. 2020; Kim et al. 2021; Cho & Kim 2022).

This watershed memory/domain knowledge and process-based modeling-related data-driven modeling, known by various terms such as physics-informed machine learning, hybrid machine learning, and theory-based machine learning, offers several benefits, including improved physical consistency, interpretability, and prediction accuracy (Humphrey et al. 2016; Noori & Kalin 2016). However, the choice between simulating watershed processes and leveraging watershed memory for performance enhancement remains unclear, with limited research exploring their comparative effectiveness and reasoning behind the selection. In particular, residual error modeling is aimed at solely improving model performance with little possibility of providing uncertainty sources. It is crucial to acknowledge that the simulation of watershed processes can introduce complexities into the process of streamflow prediction. In contrast, watershed memory-related techniques such as baseflow analysis provide a comparatively more straightforward approach. The disparity in complexity highlights the need for thorough reasoning behind technique selection, further comparative research, and examining alternatives within the framework.

This study addresses this gap by investigating the relative effectiveness of simulating hydrological processes and employing watershed memory techniques for improving data-driven streamflow prediction. We explored the efficacy of the baseflow index in enhancing prediction accuracy, a novel approach not previously investigated in this context. Additionally, we used watershed memory for the calibration and validation of process-based models, highlighting its potential for wider application. Most importantly, we tackled the critical question: can hybrid models conquer the ever-growing threat of extreme events in streamflow prediction where traditional methods falter? The findings will provide valuable insights for researchers and practitioners in the field of hydrological modeling, informing the selection of suitable approaches for data-driven streamflow prediction and contributing to the advancement of hybrid modeling approaches. This paper is structured as follows. The first section provides an overview of streamflow improvement techniques utilizing watershed memory and residual error modeling. Next, the study regions and dataset are described in detail. Subsequently, the methods section outlines the models and approaches employed in the study. The results and discussion section presents the key model outputs, followed by a concluding section that summarizes the main findings and their implications.

Watershed memory quantifies the retention and release of water within a watershed over time. It reflects the duration for which past climate, hydrogeological features, and watershed characteristics influence the current hydrological response. Conceptualizing streamflow into quick flow, interflow, and baseflow components helps understand this dynamic (Tallaksen 1995; Duncan 2019).

Baseflow, distinguished by its persistence and sustained nature within streamflow, primarily originates from groundwater and delayed sources (Hall 1968; Lim et al. 2005; Duncan 2019; McMahon & Nathan 2021). Understanding baseflow dynamics is crucial for maintaining the delicate balance between healthy ecosystems, reliable water supplies, and clean water, with applications ranging from assessing drought risks to calibrating hydrological models (Tallaksen 1995; Brutsaert 2008; Eckhardt 2008). Recognizing its significance, hydrologists and hydrogeologists have dedicated over a century to studying baseflow, gaining valuable insights into aquifer properties, and improving water resource management (Lim et al. 2005; Thiesen et al. 2019; McMahon & Nathan 2021).

BFI reflects how watershed features, geology, and land use impact water storage (Bloomfield et al. 2009; Van Loon & Laaha 2015; Sutanto & Van Lanen 2022). Watershed memory is typically represented by BFI and baseflow recession constant (Sutanto & Van Lanen 2022; Gu et al. 2023). Studies also show a positive link between this memory and accurate streamflow prediction (Harrigan et al. 2018; Girons Lopez et al. 2021).

Several data-driven studies leverage baseflow separation to enhance streamflow predictions. Corzo & Solomatine (2007) demonstrated that even traditional methods, such as the constant slope approach, can enhance simulation accuracy, with further improvement achieved using optimized baseflow filtering equations. Similarly, Zemzami & Benaabidate (2016) found that recursive digital filters outperformed simpler alternatives when paired with artificial neural networks (ANN). Two main methods utilize baseflow separation: (1) baseflow as a predictor: integrate baseflow with other inputs (Chen et al. 2021; Tongal & Booij 2018, 2022). (2) Separate models for baseflow and excess flow: build individual models for each component, combining results for prediction (Corzo & Solomatine 2007; Isik et al. 2013; Taormina et al. 2015). While both approaches demonstrably improve forecasting, comparative studies and clear selection guidance are lacking, presenting an interesting research avenue.

The hydrological model residual-based streamflow prediction enhancing framework has been applied in two ways: (1) data-driven models trained solely on process-based model errors, and (2) residual errors as additional input variables (Tian et al. 2018; Kassem et al. 2020; Sikorska-Senoner & Quilty 2021; Cho & Kim 2022). A pivotal consideration in the realm of hydrological modeling hybridization involves the impact of calibrating process-based models before their integration with machine learning frameworks. This issue was addressed by Shen et al. (2022), who evaluated the predictive performance of both calibrated and uncalibrated versions of the PCRaster Global Water Balance (PCR-GLOBWB) model when coupled with random forest (RF). Their findings revealed significant enhancements in accuracy for both configurations, signifying the substantial potential of residual error correction within this hybridization paradigm.

The Gapcheon and Chogang watersheds benefit from readily available, up-to-date streamflow and weather observations. Conversely, the Melka Kunture watershed presents a data limitation, with complete observations accessible only until the year 2013. Similar data availability constraints are evident in other studies focusing on this region, with model simulations and observed data restricted to this period (e.g., Shawul & Chakma 2020; Birhanu et al. 2021; Mitiku et al. 2023).

The SWAT model was used to develop hydrological models for the Gapcheon and Chogang watersheds, which comprise 26 and 27 subbasins and 1,295 and 1,199 HRUs, respectively. In the Gapcheon watershed, over 56% of the area exhibits a slope of less than 25%. For HRU generation in both watersheds, a threshold option of 7/7/7% for land use/soil/slope was employed. Approximately 73% of the total area in both watersheds is covered by forest. In the Chogang watershed specifically, 44% of the terrain features a slope of less than 25%.

Melka Kunture watershed had around 37 subbasins, and the HRU was created using a 3/3/3% Land use/Soil/Slope threshold. Around 86% of the Melka Kunture watershed is used for rainfed agriculture.

The SWAT model was calibrated and validated for the selected parameters using observed streamflow data from the outlets of each watershed (Table 2). Sequential uncertainty fitting version 2 (SUFI-2) was applied within the SWAT-calibration and uncertainty program (SWAT-CUP) user interface (Abbaspour et al. 2007; Abbaspour 2011) for calibration and validation. For each watershed, the calibration and validation periods were as follows:

• Gapcheon watershed: calibration: 2002–2012; validation: 2013–2020

• Chogang watershed: calibration: 2006–2014; validation: 2015–2020

• Melka Kunture: calibration: 2000–2007; validation: 2008–2013

The calibration range for the baseflow recession factor (ALPHA_BF.gw) was determined based on baseflow recession analysis results and set to around ±30% of those values. For other parameters, the initial ranges were determined systematically using published values (Shawul & Chakma 2020; Lee et al. 2023b), expert knowledge, and sensitivity simulations exploring parameter space and parameter types.

The recession analysis and BFI_max were done to separate baseflow from streamflow. The recession constant for the Gapcheon watershed was 0.978, with an average BFI_max of 0.426. For the Chogang watershed, the recession constant was 0.977, and the BFI_max was 0.344. In the Melka Kunture watershed, the recession constant was 0.959, and the BFI_max was 0.475.

Long short-term memory (LSTM) (Hochreiter & Schmidhuber 1997) is a type of recurrent neural network (RNN) designed to excel at capturing long-term dependencies in sequential data. Unlike traditional RNNs, which suffer from the vanishing gradient problem, LSTMs employ a memory cell, gates, and a complex gating mechanism to manage the flow of information. These gates allow LSTMs to selectively retain or discard information as needed, enabling them to handle long-term dependencies effectively.

LSTM has gained significant traction in hydrology for its ability to predict short- and long-term hydrological variables (Shen 2018; Kim et al. 2021; Xie et al. 2022). Detailed explanations of LSTM's inner workings are provided by numerous researchers, including Kim et al. (2021) and Kratzert et al. (2018).

This study used an LSTM layer with 50 units and the Adam optimizer with a learning rate of 0.001. The model was trained for 60–700 epochs with a batch size of 32–128. A dropout rate of 0.1 was employed for regularization. The architecture included multiple dense layers with varying numbers of units (256, 512, 512, 64) and rectified linear unit (ReLU) activation functions. These hyperparameters, including the LSTM units, learning rate, batch size, number of epochs, dropout rate, and dense layer architecture, are pivotal in influencing the model's ability to capture temporal patterns in the input data and can be fine-tuned to optimize prediction performance.

Section 2 provides an overview of two widely adopted hybridization techniques for streamflow prediction: residual error-based and watershed memory-based methods. While the former demonstrably increases prediction accuracy, it demands substantial computational resources by executing all process-based modeling steps to extract residual errors for subsequent machine learning analysis. This translates to increased cost and time investment in streamflow prediction. Conversely, watershed memory-based approaches offer relative simplicity but lack clear justifications for their selection over residual error methods.

Furthermore, while both process-based models and data-driven methods struggle with extreme events (Zhang et al. 2018; Tan et al. 2020; Yifru et al. 2024), research lacks clarity on which specific hydrological extremes benefit from hybridized approaches. Therefore, this study endeavors to address three fundamental questions regarding the two extensively employed hybridization techniques.

The methodology involves three key steps. First, baseflow recession analysis was employed to calibrate and validate the SWAT model, ensuring accurate simulation of baseflow contributions to streamflow. Next, baseflow, baseflow index, and SWAT model residuals were utilized as additional inputs to train an LSTM model, which aims to improve streamflow prediction accuracy. In this framework, four modeling approaches were investigated. The first model utilized an LSTM model with weather and observed streamflow data for prediction. In the second hybrid model, Bflow-LSTM, baseflow data joined weather and streamflow data as inputs. The third method, BFI-LSTM, employed an LSTM model with baseflow index, weather, and streamflow data. The fourth, SWAT-LSTM, utilized an LSTM to predict and update residual errors using weather and SWAT model residual errors. For all watersheds and models, data were divided into 80% for training and 20% for testing. Furthermore, 20% of the training data were allocated for cross-validation, further optimizing model performance.

Three key metrics were used to assess model performance: Nash–Sutcliffe efficiency (NSE) (Nash & Sutcliffe 1970), percent bias (PBIAS) (Sorooshian et al. 1993), and coefficient of determination (R²) (Equations (5) and (6)). NSE measures the model's ability to capture the observed variability, offering a comprehensive assessment of its overall performance. PBIAS reveals whether the model overestimates or underestimates streamflow, highlighting potential biases in the predictions. R² evaluates how well the model explains the observed variations in streamflow, providing insight into its fit and explanatory power.

In Equations (5) and (6), Q_i and S_i are the observed and computed data for the ith day, n is the length of data considered, and is the mean of n observed data, respectively.

While traditional model performance indices offer a general assessment of overall model accuracy, they often fail to reveal specific limitations in predicting critical flow regimes like peak and low flows. To address this, we employed flow duration curves (FDCs) to visually evaluate the performance of all modeled outputs and observed flow. This approach leverages the inherent sensitivity of FDCs to changes in flow frequency and magnitude, allowing us to identify potential biases in model predictions at both extreme and intermediate flow levels. To specifically assess model performance at extreme events, we zoomed in on the far ends of the FDC plots, enabling a close-up examination of peak and low flow predictions.

Based on the widely used model performance rating by Moriasi et al. (2007), the Gapcheon watershed model achieved a ‘good’ rating based on its NSE values (Table 3). However, its performance varied between calibration and validation periods, receiving ‘satisfactory’ and ‘unsatisfactory’ ratings, respectively. In contrast, the Chogang watershed model's performance was ‘unsatisfactory.’ As discussed in other studies (Tigabu et al. 2023), the observed high inter-annual flow variability in this Korean watershed might be a contributing factor to the SWAT model's underperformance. The Melka Kunture watershed model exhibited the best performance, achieving a ‘generally good’ rating.

Across study regions, performance differences between the standalone SWAT and LSTM models were negligible. The hybrid models demonstrated superior predictive accuracy compared to the standalone LSTM model. Notably, the Bflow-LSTM and BFI-LSTM models significantly outperformed the weather and streamflow-based LSTM predictions. This finding underscores the critical importance of incorporating watershed memory, particularly through baseflow and baseflow index, to enhance model robustness and ensure consistent performance across training and testing phases.

The hybrid model's performance in South Korean watersheds exhibited an NSE value range of 0.79–0.99 during training and 0.55–0.98 during testing (Table 4). Conversely, in the Ethiopian watershed, all tested hybrid models achieved an NSE value of 0.99 during training, with values ranging from 0.98 to 0.99 in the testing period. Across all watersheds, Bflow-LSTM consistently outperformed other models.

While the standalone LSTM served as the primary benchmark for evaluating hybrid model performance, comparisons with the standalone SWAT model further revealed significant improvements achieved by the SWAT residual error-based hybrid models. This enhancement was particularly noteworthy in the Korean watersheds, where the standalone SWAT model exhibited limitations. The hybrid SWAT-LSTM model demonstrably improved core streamflow prediction metrics (NSE and PBIAS) in the Korean watersheds.

Despite significant streamflow prediction improvements in the Korean watersheds compared to the standalone SWAT model in the region, the hybrid SWAT-LSTM achieved near-perfect performance in the Melka Kunture watershed. This exceptional outcome might be directly linked to the standalone SWAT model's initial performance in this specific case. While other studies (Shen et al. 2022) suggest equivalent performance improvements from hybridizing machine learning models regardless of process-based model calibration, our findings hint at a potential influence of calibration performance on hybrid model outcomes. Further investigation is warranted to explore this possible connection.

The Bflow-LSTM model significantly outperformed other models in peak flow prediction across all study watersheds. Interestingly, the SWAT model achieved performance comparable to the hybrid modeling approaches at low-flow prediction. This finding aligns with previous studies, such as Kim et al. (2021), who reported that machine learning models often excel at peak flow prediction while process-based models are better suited for low-flow simulations. Notably, each model exhibited significantly different performance at extremely low-flow prediction, with the watershed memory-based models consistently overestimating extreme low flow in all watersheds.

Recent studies (Vinuesa et al. 2020) have underscored the potential of artificial intelligence in advancing various sustainable development goals (SDGs). The environmental-related SDGs are SDG 13 (climate action), SDG 14 (life below water), and SDG 15 (life on land) (United Nations Development Programme 2016). This research aligns directly with these broader initiatives, illuminating the performance of various modeling approaches in capturing extreme events. Improved prediction of low flows enables more effective management of water resources during droughts, ensuring equitable access to safe water for all. Furthermore, understanding how models handle extreme events facilitates the development of strategies for flood preparedness and mitigation, safeguarding communities and infrastructure from the impacts of climate change.

The focus on the popular residual error approach in this study, while motivated by its widespread application, restricted the exploration of other promising hybrid techniques. This limited scope prevented an in-depth investigation of incorporating additional variables, feeding process-based model outputs directly into machine learning algorithms, and even replacing specific modules within the models themselves (Bhasme et al. 2022; Cho & Kim 2022; Feng et al. 2022; Liu et al. 2022; Yu et al. 2023). Additionally, the study solely explored baseflow-based memory techniques, overlooking the potential of other intermediate process outputs such as evapotranspiration that could further enhance model performance. Therefore, this study can serve as a valuable springboard for future research to delve into these untapped avenues.

Unlocking the full potential of hybrid hydrological models demands investigating promising alternatives beyond the residual error approach. Can incorporating residual error as a predictor refine process-based simulations, or should it be included as an input variable? Similarly, does a modular approach using watershed memory terms, encompassing both surface runoff and baseflow, outperform individual variable inputs? Evaluating the efficacy of other deep learning models within this framework also presents an intriguing avenue for future research. Answering these seemingly simple questions will unlock a new era of efficiency and accuracy in hybrid machine learning prediction studies, paving the way for more robust and adaptable hydrological models.

This study delved into the efficacy of two prevalent streamflow prediction hybridization techniques across two geographically and hydrometeorologically diverse regions: residual error-based and watershed memory-based methods. While the former demonstrably elevates accuracy by capitalizing on comprehensive process-based modeling, it incurs substantial computational expenses. Conversely, watershed memory-based approaches present a computationally efficient alternative, though their competitive edge against the more data-intensive residual error methods and the rationale for selecting one over the other remains unclear. To bridge this knowledge gap and inform best practices, this study investigates three key questions:

1. Which hybridization methodology demonstrates superior overall performance?

2. Do these techniques significantly enhance forecasts of specific hydrological extremes (low flow, peak flow)?

3. Can alternative watershed memory terms surpass the efficacy of the traditional baseflow data-based hybridization approach?

Employing a rigorous methodological framework incorporating baseflow analysis, watershed process modeling using SWAT, and development of standalone and hybrid LSTM models, our study yielded key insights. First, the Bflow-LSTM model, leveraging baseflow data, consistently outperformed all other investigated methods across diverse watersheds and metrics (NSE, PBIAS, and R²) during training and testing. Notably, it significantly improved extreme events predictions, particularly peak flows, relative to the standalone LSTM. Second, while overall performance remained modest, the standalone SWAT model showed comparably good performance at low-flow events. This highlights the continued relevance of process-based models in capturing low-flow dynamics. Third, our exploration of alternative memory terms yielded nuanced results. Although the BFI-LSTM model, which utilizes the baseflow index as memory, showed promising performance, it did not consistently surpass the Bflow-LSTM model across all watersheds and metrics.

The choice between watershed memory and residual error-based hybrid models may depend on the specific study focus. In the context of drought-related or low-flow studies, the standalone SWAT model may be a well-suited tool. However, for peak flow or flood events, the superior performance of the Bflow-LSTM model makes it a valuable tool for hydrological studies and disaster management applications.

This study was supported by the Surface Soil Conservation and Management (SS) projects, funded by the Ministry of Environment (MOE) of Korea, under grant number 2019002820003, and by a National Research Foundation of Korea (NRF) grant, which is funded by the Korea government (MSIT), with the grant number 2022R1F1A1073748.

Data cannot be made publicly available; readers should contact the corresponding author for details.

The authors declare there is no conflict.