https://orcid.org/0009-0000-8090-2546
ABSTRACT
The study compares two approaches in urban hydrology modelling: distributed models, which require detailed knowledge of sewer networks, and conceptual models using artificial neural networks that only need reliable data. The research specifically evaluated a Nonlinear AutoRegressive eXogenous (NARX) neural network model against a traditional distributed model for rainfall/discharge overflow prediction. Both models underwent identical calibration and validation processes. Results demonstrated that the NARX model achieved similar performance levels to the distributed model across various metrics, including overflow volume, peak discharge, and different efficiency measures (Nash–Sutcliffe efficiency, logarithmic Nash–Sutcliffe Efficiency, and Kling–Gupta Efficiency). Analysis of the 20 largest overflow events showed that the neural network accurately predicted overflow volumes in 75% of cases, compared to 60% for the distributed model. Performance criteria were met in nine out of 20 events for the neural network and 11 out of 20 for the distributed model. The study concludes that conceptual models, which are easier and more cost-effective to implement, can perform comparably to traditional distributed models in urban hydrology applications.
HIGHLIGHTS
Combined sewer overflow modelling.
Comparison between the distributed model and the Nonlinear AutoRegressive eXogenous neural network model.
Modelling performance analysis.
INTRODUCTION
Digital modelling applied to urban hydrology is now widely used in research programmes and operational settings, especially for conducting diagnostic studies and developing master plans for sewer systems. Engineering companies that perform these services often resort to rainfall–runoff modelling to quantify the system's response to rainfall events. This analysis helps determine the most appropriate strategies for protecting people and property and controlling the system's impact on the receiving environment. The range of studies in which models are used has greatly expanded in recent decades. In addition to master plans and diagnostic studies, models are, for example, used in long-term simulations to determine the impact of climate change on urban systems (Pons et al. 2022). Operational uses have also emerged in the context of predictive systems for real-time control or to fill data gaps in a reporting context that may be regulatory. Depending on the needs, the knowledge of the system, and the budget available to build a model, certain approaches will be promoted.
If we take the last example of regulatory reporting, the model is now accepted in Europe to produce overflow data from a sewage system when overflow structures are not instrumented or when instrumentation has periods of failure. The objective of constructing overflow information is relatively modest compared to all the possibilities offered by modelling. Also, communities wishing to implement this approach need simple models that are easy to build and inexpensive and require minimal prerequisites regarding precise knowledge of the infrastructure. Machine learning models can meet most of these criteria. However, for a model to be accepted by authorities in the context of regulatory data production, it must also be sufficiently accurate. The objective of this paper is to compare a classic approach using a conceptual model with a machine learning approach in order to compare their performances and determine whether or not, in a context where knowledge and resources are limited, the second type of model can present an interesting alternative from an operational point of view.
There are several categories of models. Depending on the chosen model type, the prerequisites for knowledge about the studied system and available data are not the same, and the implementation difficulties are not all equivalent. Hydrological models are usually classified based on the representation of processes, spatial scale considered, or the temporal representation of phenomena (Ambroise 1999; Singh & Woolhiser 2002; Hingray et al. 2009; Jay-Allemand 2020).
Software tools for building distributed conceptual models are widely used in operational settings today, but their use requires certain prerequisites that can act as barriers to their adoption. The two main obstacles are, firstly, knowledge of the characteristics of the sewer network, including elevation information for pipes and ground, and secondly, the availability of distributed measurements across the catchment to enable the distributed model calibration.
One way to overcome the need for a precise understanding of the studied system's structure is to develop empirical models, such as neural network models, which require data related to the system's operation but not specific knowledge about the description of its structures or organization. Neural networks are artificial intelligence systems organized into layers that include a variable number of neurons. The user presents inputs to the network, variables governing the phenomenon, and outputs the data to be simulated. The network learns how to connect inputs to outputs through an optimization algorithm (Hecht-Nielsen 1990).
The use of neural networks to model hydrological processes began in the early 1990s (Govindaraju 2000). Since then, applications have covered a wide range of aspects in hydrology and hydraulics, including rainfall–runoff modelling (Hsu et al. 1995), river flow simulation (Campolo et al. 1999), water quality (Chen et al. 2020), groundwater levels (Coulibaly et al. 2001), velocity field downstream to an open channel junction (Sun et al. 2014), and precipitation estimation (Olsson et al. 2004). Neural network models are primarily used in hydrology due to their ease of implementation (Riad et al. 2004; Govindaraju & Rao 2013; Aichouri et al. 2015; Tanty & Desmukh 2015). For all these applications, neural networks are considered robust alternatives for simulating nonlinear phenomena. This capability of neural networks has been exploited by several authors who propose a hybrid modelling approach. It involves simulating a phenomenon using a deterministic model and applying a machine learning approach to process the error between the deterministic model and the measurement in order to improve the accuracy of the hybrid model. This hybrid approach finds its application in various fields such as sewer network flows (Vojinovic et al. 2003), tide prediction (Sannasiraj et al. 2004), or the prediction of water quality parameters (Wang et al. 2016). This approach greatly improves prediction but does not allow for dispensing with the construction of a deterministic or conceptual model as a support for modelling the phenomenon. In the specific context of urban sewer network operation, and particularly rainfall-induced overflows, few authors have been interested in the subject. Most of the referenced studies share a common emphasis on operational objectives and, as a result, focus on relatively short prediction horizons used to manage control/command processes. Some authors rely on multilayer perceptron neural networks (Kurth et al. 2008; Mounce et al. 2014; Rosin et al. 2017), and others compare various machine learning methods (Zhang et al. 2018a, b).
Regarding time series simulation, like rainfall–runoff simulations, particularly efficient neural networks are dynamic networks (Dreyfus et al. 2002; Khalil 2012). This network structure includes feedback from outputs to inputs. Thus, a dynamic network can take as input the data from the current calculation time step, as well as input data and network outputs from a certain number of preceding time steps. This architecture effectively captures the dynamics of phenomena and is considered to have ‘memory’. Hydrological parameters, such as lag time and concentration time of a watershed, illustrate the importance of this property. In the context of a rainfall–runoff model, the flow at a given time depends on the rainfall over a more or less extended period preceding the current time step. More intuitively, within a hydrograph, the flow at a given time is dependent on the flow over a set of time steps preceding the current time step.
Open-loop operation during the learning phase: the actual output values are known and used as inputs. This mode has two main advantages, namely having unbiased inputs and the ability to apply backpropagation algorithms during the learning phase, given that the architecture is feedforward in this configuration (Garnier & Young 2004).
Closed-loop operation during the simulation phase: simulated outputs are fed back into the system as inputs.
Architecture of a NARX-type neural network with one hidden layer, illustrating open-loop operation during the learning phase on the left and closed-loop operation during validation and simulation on the right. The indices t, t − 1,…, t − n correspond to the time steps for which values are considered as inputs to the network.
Architecture of a NARX-type neural network with one hidden layer, illustrating open-loop operation during the learning phase on the left and closed-loop operation during validation and simulation on the right. The indices t, t − 1,…, t − n correspond to the time steps for which values are considered as inputs to the network.
The NARX can incorporate exogenous data inputs over a defined number of time steps and integrate the simulated output results in a closed-loop manner.
The bibliography related to the use of NARX in hydrology includes authors who have focused on both urban hydrology and non-urban watershed hydrology. El Ghazouli et al. (2019) evaluated two methods for predicting rainfall-induced discharge on an urban watershed in Morocco over 30 min: a regression tree algorithm (M5 algorithm) and a NARX-type neural network. The NARX model outperformed the M5 algorithm and was chosen for real-time management of Casablanca's sanitation network to reduce overflow impacts (El Ghazouli et al. 2022). NARX models were also used by Zhang et al. (2018a) and El Ghazouli et al. (2019) for short-term control and optimization, with continuous recalibration using real-time data. Wei et al. (2020) applied a NARX model in a flood risk early warning system in Tianjin, China. They predicted rainfall based on the first 20 min of precipitation and assessed flood risks with a hydraulic model. Although interpreting their performance metrics was challenging, the NARX model was effective for rainfall prediction and flood alerts.
In a broader context, considering applications in general hydrology, the number of studies involving autoregressive networks remains limited, even though the vast majority of them attest to the performance of such networks (Thapa et al. 2020).
Continuing on the theme of hydrology, Shao et al. (2022) applied the artificial neural network (ANN) method to a natural watershed of approximately 10,000 km2 in eastern China. The authors compared the performance of a NARX-type model with that of a distributed hydrological model in rainfall–runoff simulation, considered as the reference. The authors justified their choice of NARX as an empirical model by highlighting its effective application for predicting nonlinear time series compared to other machine learning algorithms. They insist that the NARX structure allows for a physical interpretation of the studied parameter. Finally, NARX networks have faster convergence during the learning phase compared to most competing approaches. The comparison criteria chosen include the Nash–Sutcliffe coefficient (NSE), the correlation coefficient (CC), the root mean square error (RMSE), and the mean relative bias (Bias). The main conclusion is that the NARX model can satisfactorily simulate the rainfall–runoff process, accurately reproducing peak flows in terms of intensities and occurrence dates. In addition, the NARX model is capable of accounting for hydrological phenomena with complex nonlinear dynamics, as analysed through each of the four selected performance criteria (NSE, CC, RMSE, and Bias). However, the authors highlight a tendency of the NARX to underestimate the peak flow values. On the same topic, Di Nunno et al. (2021) used a NARX-type network to predict the discharges of nine rivers in an Italian watershed. The authors highlighted the strong performance of NARX over prediction horizons ranging from one to 12 months, with CCs between predictions and measurements exceeding 0.9 in both cases.
Furthermore, focusing on the rainfall–runoff relationship in an urban setting, Abou Rjeily et al. (2017) propose the use of a NARX-type network to simulate water levels in the inspection chambers of the drainage system in a district of Lille (north of France) based on rainfall. The ultimate goal of their work is to establish an early warning system for sewer overflows. A simulation phase is then carried out based on the impact of simulated design rainfall on the network using a distributed model. Comparing water levels provided by each model for five manholes and three rainfalls with different return periods reveals that the NSE consistently exceeds 0.8 in almost all simulations. A particularly interesting aspect of this study, though not discussed by the authors, is that the return periods used in the simulation phase (one year, two years, and five years) are not definitively represented in the observational dataset, particularly for the 5-year return period. This observation highlights the extrapolation capabilities of this type of network in the case addressed by the authors.
This literature review confirms the relevance of NARX in hydrological modelling. In addition, the structure of NARX-type networks also allows for predictions over a large number of future time steps (Menezes & Barreto 2008) as long as exogenous inputs are available to feed the model. In this context, the NARX is used in a closed-loop configuration for prediction. Despite this, it is noted that the existing literature lacks studies on the use of this type of neural network in urban environments for performing long-term simulations to assess overflow volume and frequencies.
The comparison between the performance of different artificial intelligence models has been the subject of several studies in hydrology. However, the comparison of performance between an artificial intelligence model and a distributed model has been little studied (Shao et al. 2022). When both types of models are considered, the distributed model is often used as a data generator for training and testing the neural network (Abou Rjeily et al. 2017). Therefore, this work focusing on combined sewer overflows (CSOs) presents an original character.
We propose here to perform the construction, calibration, and validation of two types of models: an empirical neural network model based on the NARX approach and a ‘traditional’ distributed model created using the dedicated software commonly used in consulting engineering. The performances of these two models will be compared to determine whether the empirical alternative based on a NARX-type neural network is acceptable for simulating CSO volume.
METHODS
Study site
This CSO generates the largest discharge volumes, averaging 450,000 m3 annually, corresponding to approximately 45% of the total annual volume discharged by the entire system.
This CSO has been extensively studied, and the discharge measurement is deemed reliable (Claro Barreto 2020). Presently, the confidence level in the measurement is very high, which is not always the case for measurements taken at other CSOs.
Schematic of the Gambetta CSO with the upstream collector on the right and the downstream flow on the left regulated through a sluice gate. The overflow discharge occurs through the two circular collectors (marked left and right) (adapted from Claro Barreto (2020)).
Schematic of the Gambetta CSO with the upstream collector on the right and the downstream flow on the left regulated through a sluice gate. The overflow discharge occurs through the two circular collectors (marked left and right) (adapted from Claro Barreto (2020)).
Dataset
Information required for developing both the neural network and the distributed model comprises flow values at the network's points of interest and the corresponding rainfall data. Note: All the data used in this study are the property of the local authority.
Flow measurements have been continuously recorded at the Gambetta CSO since 2009, in accordance with regulatory requirements. However, as the measurements only became reliable in late 2017 (Claro Barreto 2020), we only considered post-refinement data for this study. Furthermore, exchanges with the local authority revealed network modifications in the recent period. The construction of a storage capacity upstream of the Gambetta CSO in June 2020 altered rainfall contributions at the Gambetta Overflow. As a result, only discharge measurements between January 1, 2018 (the date of measurement refinement) and June 30, 2020 (the date when the network configuration was modified) were considered. All flow data were recorded at a time step of 5 min.
Flow measurement downstream of Bourg-lès-Valence (1)
Flow measurement in the Valence branch upstream of the Gambetta CSO (2)
Measurement of the discharged flow at the Gambetta CSO (3)
Flow measurement upstream of the wastewater treatment plant (WWTP) corresponds to the North sewer, which collects effluents from Valence and Bourg-lès-Valence cities (4).
Model coverage with the selected calibration points. The orange sector corresponds to the Bourg-lès-Valence catchment (1), and the blue sector to the Upstream-Gambetta basin (2). At the Gambetta CSO (3), effluents from the orange, blue, and northern parts of the green sectors are collected. Lastly, the Upstream WWTP (4) point receives effluents from all orange, blue, and green sectors. The green triangles correspond to the locations of the rain gauges.
Model coverage with the selected calibration points. The orange sector corresponds to the Bourg-lès-Valence catchment (1), and the blue sector to the Upstream-Gambetta basin (2). At the Gambetta CSO (3), effluents from the orange, blue, and northern parts of the green sectors are collected. Lastly, the Upstream WWTP (4) point receives effluents from all orange, blue, and green sectors. The green triangles correspond to the locations of the rain gauges.
The rainfall time series used for this study was obtained from the local authority rain gauges. In terms of rainfall depth, 2018 was a rather wet year with a total rainfall depth of 905 mm rainfall for the year at the Belle Meunière rain gauge (compared to a median value of 834 mm for the period 2010–2021), 2019 was a relatively dry year with a total rainfall depth of 771 mm, and 2020 was a very dry year with a total rainfall depth of 554 mm. Rainfall data were available at a 6-min time step on the three rain gauges in the neighbourhood of the Gambetta CSO catchment. Rainfall is a crucial parameter in urban hydrology (Ballinas-González et al. 2020), and failing to account for its spatial and temporal heterogeneity introduces a source of uncertainty (Fraga et al. 2019). Indeed, the incorporation of data from three rain gauges enhanced the model calibration. According to Schilling (1984), three rain gauges for a 15 km2 catchment (approximately the size of the studied system) allow for obtaining satisfactory results using a rainfall–runoff model with a maximum uncertainty on the watershed flow of approximately 20–30%. Data preparation involved converting rainfall data from a 6-min time step to a 5-min time step to achieve consistency in time intervals between rainfall and discharged flow data. While this step may not be essential for calculation with the distributed model, it was needed for the neural network.
Building the distributed model
The data necessary for designing the model were derived from the local authority's GIS and an existing model of the catchment. The population was recalculated based on public data, and the average slope of sub-catchments and various types of surfaces (roof, road, and pervious surface) were extracted from the French public database.
The final model, built using InfoWorks-CS™ software, covered an area of 1,574 hectares, divided into 154 sub-catchments, with 33% of the surface being impervious (mainly roofs and roads). The modelled network included 269 calculation nodes and a total pipe length of 48 km. The total population attached to the model was 97,367 inhabitants distributed across all sub-catchments in proportion to their area.
The method for designing a distributed model requires calibrating the model in dry weather and then in wet weather. While dry weather calibration is not a significant challenge, as the discharge profile is relatively stable, calibration and validation in wet weather are the real challenges. Wet weather calibration is achieved by adjusting the parameters of runoff models applied to different urban surfaces. In our case, the runoff model used for impervious surfaces was a linear reservoir model with a lag time dependent on the characteristics of the urban catchment and the rainfall intensity (Desbordes 1987). The final runoff coefficient selected was 0.3 for roads and 0.85 for roofs. Runoff on pervious surfaces was described by a Horton model (Horton 1941), with an initial infiltration capacity of 76 mm/h and a limiting infiltration capacity of 12.7 mm/h. For the hydraulic part, the flow propagation in the pipes was ensured through a complete resolution of the Barré de St-Venant equations (Cunge & Wegner 1964).
Calibration and verification phases were conducted over continuous periods with a calculation time step of 5 min.
Building the NARX neural network
The parameters involved in building a NARX-type neural network are related to the network's structure, specifically the number of neurons in the hidden layer and the considered time delay. The second parameter corresponds to the number of past time steps considered as input to produce the output at the current time step. Tuning the network involves defining the optimal values for these two parameters.
The performance of a learning neural network is generally good, and analysing indicators in detail on the training data is not particularly informative. We therefore focused on a more in-depth analysis of the network's performance on the verification dataset. The calibration phase involved searching for the network configuration with the best performance on the validation dataset. For the design of the network, the conducted simulations cover combinations of values: five, 10, and 15 neurons in the hidden layer and six, 12, 18, and 24 timesteps for the delay. To assess the repeatability of the network construction step, each configuration was generated 30 times successively; a value chosen deliberately to maintain a computation time compatible with the completion of this work. Testing three different values for the number of neurons in the hidden layer combined with four possible values for the delay and 30 repetitions for each configuration resulted in a total of 360 generated networks, equivalent to approximately 9 h of computation on a standard computer.
As for the distributed model, the calibration and verification phases were conducted over continuous periods with a calculation time step of 5 min.
Performing calibration, validation, and verification
Splitting the datasets
Distribution of daily discharged volumes measured at the Gambetta CSO, by year, for days with positive overflow only. Note: for 2020, the dataset only includes measurements up to June.
Distribution of daily discharged volumes measured at the Gambetta CSO, by year, for days with positive overflow only. Note: for 2020, the dataset only includes measurements up to June.
The selection of training and validation periods must satisfy two conditions as closely as possible:
The calibration/validation (training) phase should cover the widest possible range of discharged flow values.
The datasets chosen for each of the two phases should be as similar as possible.
We chose a calibration/validation (training) period, which covered 2019 and the first half of 2020, while the verification period was the year 2018. This initial data split is the same for both types of models: the neural network and the distributed model, allowing for a comparison of the performance of the two models under verification conditions.
Building the NARX model requires another splitting procedure involving randomly splitting the dataset into three parts. The three resulting sets are used in each of the following steps:
The first group is used for training and enables the automatic adjustment of connection weights between neurons.
The second set of data is the validation set. It is used to measure the network's generalization and to stop training when performance stops to improve. This set allows testing the learning algorithm's performance.
The last set of data is used for testing and does not affect learning. It is, therefore, an independent measure of the network's performance at the end of training.
Data split for the distributed model and the NARX model. The year 2018 is dedicated to verification for both models. The years 2019 and 2020 are dedicated to calibration for the distributed model, which is divided into three parts to perform calibration, validation, and testing for the NARX model.
Data split for the distributed model and the NARX model. The year 2018 is dedicated to verification for both models. The years 2019 and 2020 are dedicated to calibration for the distributed model, which is divided into three parts to perform calibration, validation, and testing for the NARX model.
This construction method involves an initial data split by randomly selecting values from the dataset and assigning them to each category. Therefore, isolated time steps are assigned to each dataset, and it is impossible to have complete events related to a specific phase (learning, validation, or testing). Consequently, this operating mode does not allow testing the network's ability to reproduce a flow generated by a rainfall event as a whole. This approach is quite different from modelling practices adopted for distributed conceptual rainfall–runoff models, where complete rainfall events or even continuous series of rainfall events are used for calibration and validation phases, respectively.
Verification: evaluation criteria for performance
The performance evaluation of the calibration and verification stages was performed by comparing the model's output results with measurements, considering a continuous prediction over the time series. Errors were assessed based on annual total overflow volumes over the year 2018. For the most intense CSO events (20 events corresponding to approximately 85% of the discharged volume in 2018), a detailed comparison of the observed CSO hydrographs with the results from each of the two models was conducted. Two types of criteria are commonly used in hydrology to determine the performance of the models. On one hand, the overflow volumes and the peak flow values provide a general estimate of the model's performance. On the other hand, the NSE, the logarithmic NSE (logNSE), and the Kling–Gupta Efficiency (KGE) allow for an assessment of the quality of the hydrograph shape reproduction (Nash & Sutcliffe 1970; Bertrand-Krajewski et al. 2000; Schaefli & Gupta 2007; Aguilar et al. 2016; Chadalawada & Babovic 2019).
The performance of both the models was evaluated during the verification period in 2018. Each overflow episode was assessed against each criterion, and an overall assessment was made of the number of episodes that met the selected performance criteria. In this study, results were considered of very good quality if the difference between simulation and measurement was less than 15% for volume and less than 30% for peak flow. For hydrogram fitting, we considered a very good quality if NSE was greater than 0.75, as recommended by Moriasi et al. (2007, 2015). In the same way, a value of 0.8 or higher has been selected for the logNSE (Pushpalatha et al. 2012), and a value of 0.75 or higher is considered for KGE (Knoben et al. 2019). In addition, the computational time required for simulation was considered in the performance evaluation.
RESULTS AND DISCUSSION
Results over the verification period
For each combination of the number of neurons in the hidden layer and the delay value, 30 different networks were constructed. The results obtained from the 30 iterations of each configuration were filtered to retain only the networks that show a difference less than or equal to 15% in volumes and an NSE greater than or equal to 0.75 for the entire year 2018. The configuration that presents the best results corresponds to 10 neurons in the hidden layer and a delay of 6 min. Out of the 30 iterations, 19 satisfy those criteria for volume difference and NSE. Finally, we selected the iteration that displays the highest performance in terms of volume difference for the most significant episodes of 2018 (mean volume difference of 15% and mean NSE of 0.81).
Over the entire year 2018, the measured overflow volume was 404,098 m3. The neural network simulated a total overflow of 462,599 m3, which represents a deviation of +14% compared to the measurement. The distributed model simulated an overflow of 412,550 m3, representing a deviation of +2% from the measured value. Across all 20 main CSO events, the 15% maximum deviation criterion for volume was met in 75% of the events for the neural network and 60% for the distributed model. For peak flow, the results showed that 70 and 80% of the events met the 30% maximum deviation criterion for the neural network model and the distributed model, respectively. The NSE criterion exceeding 0.75 was met in 75% of the events for the neural network and in 60% of the events for the distributed model. The logNSE criterion is the most disparate, as it exceeds 0.8 in only 10% of cases for the neural network, compared to 80% for the distributed model. Finally, the KGE exceeds 0.75 for 55% of the events simulated by the neural network and 60% of the events simulated by the distributed model. It should be noted that for each of the criteria, the highest limit allowing to characterize a very good performance was retained.
All five criteria were met in two events out of 20 for the neural network and in eight events out of 20 for the distributed model. If we consider the achievement of only four out of the five criteria, nine events out of 20 meet four criteria for the neural network and 11 events out of 20 for the distributed model. These results highlight comparable performances for both models, except in terms of peak flow, where the neural network tends to underestimate values (which is in line with the findings of Shao et al. (2022)). Additionally, the neural networks show poor performance for logNSE. This is related to the fact that the neural network regularly simulates non-zero flow values even when the recorded flow is zero. These values are generally low and have little impact on the calculation of the overflow volume. However, they significantly degrade the logKGE. Table 1 shows a general overview of the models' performances regarding the selected criterion from section 2.5.2.
Performance summary for the two types of models and all selected criteria
| NARX | Distributed model | |
|---|---|---|
| Annual CSO volume | + 14% deviation from measured | + 2% deviation from measured |
| Event CSO volume (<15% deviation from measured) | Met for 75% of the 20 biggest events | Met for 60% of the 20 biggest events |
| Peak flow (<30% deviation from measured) | Met for 70% of the 20 biggest events | Met for 80% of the 20 biggest events |
| NSE (>0.75) | Met for 75% of the 20 biggest events | Met for 60% of the 20 biggest events |
| logNSE (>0.80) | Met for 10% of the 20 biggest events | Met for 80% of the 20 biggest events |
| KGE (>0.75) | Met for 55% of the 20 biggest events | Met for 60% of the 20 biggest events |
| All five criteria | 2/20 | 8/20 |
| Four criteria out of five | 9/20 | 11/20 |
| NARX | Distributed model | |
|---|---|---|
| Annual CSO volume | + 14% deviation from measured | + 2% deviation from measured |
| Event CSO volume (<15% deviation from measured) | Met for 75% of the 20 biggest events | Met for 60% of the 20 biggest events |
| Peak flow (<30% deviation from measured) | Met for 70% of the 20 biggest events | Met for 80% of the 20 biggest events |
| NSE (>0.75) | Met for 75% of the 20 biggest events | Met for 60% of the 20 biggest events |
| logNSE (>0.80) | Met for 10% of the 20 biggest events | Met for 80% of the 20 biggest events |
| KGE (>0.75) | Met for 55% of the 20 biggest events | Met for 60% of the 20 biggest events |
| All five criteria | 2/20 | 8/20 |
| Four criteria out of five | 9/20 | 11/20 |
The 20 main overflow episodes, covering 85% of the annual overflow volume at the Gambetta CSO, are detailed in Table 2 for the general metrics (overflow volume and peak flow) and in Table 3 for the three metrics related to hydrograph shapes (NSE, logNSE, and KGE).
Performance indicator values for each of the 20 most significant discharge episodes of the year 2018 (representing 85% of the total discharged volume) for both models
| Date | Overflow volume | Peak flow | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Measurement (m3) | NARX (m3) | Difference (%) | Distributed model (m3) | Difference (%) | Measurement (m3/s) | NARX (m3/s) | Difference (%) | Distributed model (m3/s) | Difference (%) | |
| 1/26/2018 | 36,999 | 34,569 | −7 | 32,297 | −13 | 2.30 | 1.70 | −26 | 1.82 | −21 |
| 3/15/2018 | 20,272 | 18,156 | −10 | 19,919 | −2 | 2.70 | 2.17 | −20 | 2.89 | +7 |
| 3/30/2018 | 24,305 | 24,206 | 0 | 23,355 | −4 | 2.21 | 1.89 | −15 | 2.21 | 0 |
| 4/4/2018 | 5,253 | 5,142 | −2 | 6,354 | +21 | 1.64 | 2.00 | +22 | 2.25 | +37 |
| 4/10/2018 | 14,726 | 15,253 | +4 | 12,702 | −14 | 2.95 | 2.02 | −31 | 2.42 | −18 |
| 4/29/2018 | 6,060 | 8,746 | +44 | 5,446 | −10 | 0.93 | 0.88 | −5 | 0.88 | −5 |
| 5/9/2018 | 3,269 | 2,806 | −14 | 5,224 | +60 | 1.32 | 0.88 | −33 | 1.68 | +27 |
| 5/13/2018 | 10,305 | 11,060 | +7 | 9,586 | −7 | 0.57 | 0.35 | −39 | 0.48 | −15 |
| 5/28/2018 | 11,808 | 6,564 | −44 | 13,379 | 13 | 2.81 | 0.84 | −70 | 3.08 | +10 |
| 6/11/2018 | 13,175 | 15,185 | +15 | 26,778 | +103 | 1.70 | 1.68 | −1 | 5.91 | +247 |
| 7/15/2018 | 7,469 | 4,311 | −42 | 11,159 | +49 | 2.14 | 1.43 | −33 | 3.82 | +78 |
| 8/13/2018 | 7,312 | 5,056 | −31 | 14,904 | +104 | 1.35 | 0.83 | −39 | 3.59 | +166 |
| 10/7/2018 | 14,565 | 16,314 | +12 | 16,814 | +15 | 1.69 | 1.46 | −14 | 2.04 | +21 |
| 10/29/2018 | 33,287 | 30,382 | −9 | 27,852 | −16 | 1.62 | 1.36 | −16 | 1.38 | −15 |
| 10/31/2018 | 21,008 | 18,234 | −13 | 14,285 | −32 | 1.72 | 1.32 | −24 | 1.30 | −25 |
| 11/6/2018 | 42,121 | 42,368 | +1 | 41,036 | −3 | 2.93 | 2.12 | −28 | 2.72 | −7 |
| 11/20/2018 | 21,085 | 21,827 | +4 | 21,711 | +3 | 2.15 | 2.00 | −7 | 2.69 | +25 |
| 11/23/2018 | 33,699 | 38,788 | +15 | 38,055 | +13 | 2.50 | 2.34 | −6 | 3.06 | +22 |
| 11/25/2018 | 8,991 | 7,109 | −21 | 6,216 | −31 | 1.49 | 1.21 | −18 | 1.17 | −22 |
| 12/19/2018 | 8,012 | 8,706 | +9 | 7,278 | −9 | 1.41 | 1.30 | −8 | 1.27 | −10 |
| Date | Overflow volume | Peak flow | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Measurement (m3) | NARX (m3) | Difference (%) | Distributed model (m3) | Difference (%) | Measurement (m3/s) | NARX (m3/s) | Difference (%) | Distributed model (m3/s) | Difference (%) | |
| 1/26/2018 | 36,999 | 34,569 | −7 | 32,297 | −13 | 2.30 | 1.70 | −26 | 1.82 | −21 |
| 3/15/2018 | 20,272 | 18,156 | −10 | 19,919 | −2 | 2.70 | 2.17 | −20 | 2.89 | +7 |
| 3/30/2018 | 24,305 | 24,206 | 0 | 23,355 | −4 | 2.21 | 1.89 | −15 | 2.21 | 0 |
| 4/4/2018 | 5,253 | 5,142 | −2 | 6,354 | +21 | 1.64 | 2.00 | +22 | 2.25 | +37 |
| 4/10/2018 | 14,726 | 15,253 | +4 | 12,702 | −14 | 2.95 | 2.02 | −31 | 2.42 | −18 |
| 4/29/2018 | 6,060 | 8,746 | +44 | 5,446 | −10 | 0.93 | 0.88 | −5 | 0.88 | −5 |
| 5/9/2018 | 3,269 | 2,806 | −14 | 5,224 | +60 | 1.32 | 0.88 | −33 | 1.68 | +27 |
| 5/13/2018 | 10,305 | 11,060 | +7 | 9,586 | −7 | 0.57 | 0.35 | −39 | 0.48 | −15 |
| 5/28/2018 | 11,808 | 6,564 | −44 | 13,379 | 13 | 2.81 | 0.84 | −70 | 3.08 | +10 |
| 6/11/2018 | 13,175 | 15,185 | +15 | 26,778 | +103 | 1.70 | 1.68 | −1 | 5.91 | +247 |
| 7/15/2018 | 7,469 | 4,311 | −42 | 11,159 | +49 | 2.14 | 1.43 | −33 | 3.82 | +78 |
| 8/13/2018 | 7,312 | 5,056 | −31 | 14,904 | +104 | 1.35 | 0.83 | −39 | 3.59 | +166 |
| 10/7/2018 | 14,565 | 16,314 | +12 | 16,814 | +15 | 1.69 | 1.46 | −14 | 2.04 | +21 |
| 10/29/2018 | 33,287 | 30,382 | −9 | 27,852 | −16 | 1.62 | 1.36 | −16 | 1.38 | −15 |
| 10/31/2018 | 21,008 | 18,234 | −13 | 14,285 | −32 | 1.72 | 1.32 | −24 | 1.30 | −25 |
| 11/6/2018 | 42,121 | 42,368 | +1 | 41,036 | −3 | 2.93 | 2.12 | −28 | 2.72 | −7 |
| 11/20/2018 | 21,085 | 21,827 | +4 | 21,711 | +3 | 2.15 | 2.00 | −7 | 2.69 | +25 |
| 11/23/2018 | 33,699 | 38,788 | +15 | 38,055 | +13 | 2.50 | 2.34 | −6 | 3.06 | +22 |
| 11/25/2018 | 8,991 | 7,109 | −21 | 6,216 | −31 | 1.49 | 1.21 | −18 | 1.17 | −22 |
| 12/19/2018 | 8,012 | 8,706 | +9 | 7,278 | −9 | 1.41 | 1.30 | −8 | 1.27 | −10 |
Grey cells highlight the values where the performance criteria are met (differences between simulation and measurement less than 15% for volume and less than 30% for peak flow).
Performance indicator values for each of the 20 most significant discharge episodes of the year 2018 (representing 85% of the total discharged volume) for both models
| Date | NSE | logNSE | KGE | |||
|---|---|---|---|---|---|---|
| NARX vs. measurement | Distributed model vs. measurement | NARX vs. measurement | Distributed model vs. measurement | NARX vs. measurement | Distributed model vs. measurement | |
| 1/26/2018 | 0.82 | 0.82 | 0.54 | 0.97 | 0.84 | 0.84 |
| 3/15/2018 | 0.92 | 0.98 | 0.65 | 0.96 | 0.74 | 0.98 |
| 3/30/2018 | 0.80 | 0.79 | 0.50 | 0.46 | 0.89 | 0.81 |
| 4/4/2018 | 0.89 | 0.70 | 0.44 | 0.95 | 0.91 | 0.58 |
| 4/10/2018 | 0.85 | 0.94 | 0.38 | 0.84 | 0.68 | 0.81 |
| 4/29/2018 | 0.87 | 0.93 | 0.61 | 0.94 | 0.84 | 0.89 |
| 5/9/2018 | 0.66 | 0.42 | 0.43 | 0.88 | 0.57 | 0.26 |
| 5/13/2018 | 0.50 | 0.89 | 0.32 | 0.85 | 0.40 | 0.78 |
| 5/28/2018 | 0.30 | 0.86 | 0.49 | 0.82 | 0.13 | 0.84 |
| 6/11/2018 | 0.78 | −2.07 | 0.94 | 0.96 | 0.88 | −0.77 |
| 7/15/2018 | 0.17 | −1.42 | 0.64 | 0.47 | 0.40 | −0.07 |
| 8/13/2018 | 0.34 | −1.09 | 0.64 | 0.75 | 0.20 | −0.48 |
| 10/7/2018 | 0.85 | 0.74 | 0.31 | 0.67 | 0.81 | 0.75 |
| 10/29/2018 | 0.80 | 0.89 | 0.33 | 0.90 | 0.74 | 0.78 |
| 10/31/2018 | 0.80 | 0.81 | 0.56 | 0.95 | 0.69 | 0.60 |
| 11/6/2018 | 0.93 | 0.95 | 0.92 | 0.99 | 0.91 | 0.95 |
| 11/20/2018 | 0.76 | 0.55 | 0.41 | 0.90 | 0.87 | 0.67 |
| 11/23/2018 | 0.80 | 0.62 | 0.70 | 0.96 | 0.79 | 0.66 |
| 11/25/2018 | 0.93 | 0.92 | 0.69 | 0.99 | 0.76 | 0.77 |
| 12/19/2018 | 0.96 | 0.94 | 0.47 | 0.98 | 0.91 | 0.89 |
| Date | NSE | logNSE | KGE | |||
|---|---|---|---|---|---|---|
| NARX vs. measurement | Distributed model vs. measurement | NARX vs. measurement | Distributed model vs. measurement | NARX vs. measurement | Distributed model vs. measurement | |
| 1/26/2018 | 0.82 | 0.82 | 0.54 | 0.97 | 0.84 | 0.84 |
| 3/15/2018 | 0.92 | 0.98 | 0.65 | 0.96 | 0.74 | 0.98 |
| 3/30/2018 | 0.80 | 0.79 | 0.50 | 0.46 | 0.89 | 0.81 |
| 4/4/2018 | 0.89 | 0.70 | 0.44 | 0.95 | 0.91 | 0.58 |
| 4/10/2018 | 0.85 | 0.94 | 0.38 | 0.84 | 0.68 | 0.81 |
| 4/29/2018 | 0.87 | 0.93 | 0.61 | 0.94 | 0.84 | 0.89 |
| 5/9/2018 | 0.66 | 0.42 | 0.43 | 0.88 | 0.57 | 0.26 |
| 5/13/2018 | 0.50 | 0.89 | 0.32 | 0.85 | 0.40 | 0.78 |
| 5/28/2018 | 0.30 | 0.86 | 0.49 | 0.82 | 0.13 | 0.84 |
| 6/11/2018 | 0.78 | −2.07 | 0.94 | 0.96 | 0.88 | −0.77 |
| 7/15/2018 | 0.17 | −1.42 | 0.64 | 0.47 | 0.40 | −0.07 |
| 8/13/2018 | 0.34 | −1.09 | 0.64 | 0.75 | 0.20 | −0.48 |
| 10/7/2018 | 0.85 | 0.74 | 0.31 | 0.67 | 0.81 | 0.75 |
| 10/29/2018 | 0.80 | 0.89 | 0.33 | 0.90 | 0.74 | 0.78 |
| 10/31/2018 | 0.80 | 0.81 | 0.56 | 0.95 | 0.69 | 0.60 |
| 11/6/2018 | 0.93 | 0.95 | 0.92 | 0.99 | 0.91 | 0.95 |
| 11/20/2018 | 0.76 | 0.55 | 0.41 | 0.90 | 0.87 | 0.67 |
| 11/23/2018 | 0.80 | 0.62 | 0.70 | 0.96 | 0.79 | 0.66 |
| 11/25/2018 | 0.93 | 0.92 | 0.69 | 0.99 | 0.76 | 0.77 |
| 12/19/2018 | 0.96 | 0.94 | 0.47 | 0.98 | 0.91 | 0.89 |
Grey cells highlight the values where the performance criteria are met (NSE greater than 0.75, logNSE greater than 0.80 and KGE greater than 0.75).
Comparison of overflow hydrographs for each of the 20 most significant rainfall events leading to overflow events of the year 2018. Measured values are represented in grey, neural network outputs in green, and distributed model outputs in red. Rainfall (blue) from one of the three rain gauges used as input for the models is represented in the reverse scale on each graph.
Comparison of overflow hydrographs for each of the 20 most significant rainfall events leading to overflow events of the year 2018. Measured values are represented in grey, neural network outputs in green, and distributed model outputs in red. Rainfall (blue) from one of the three rain gauges used as input for the models is represented in the reverse scale on each graph.
Comparison of daily discharged volumes pairwise throughout the year 2018 from left to right: NARX results against measurements (a), distributed model results against measurements (b), and distributed model results against NARX results (c). The regression line, its equation, and the CC are specified on the graphs, along with the envelope at ±15% represented by the pink dashed lines.
Comparison of daily discharged volumes pairwise throughout the year 2018 from left to right: NARX results against measurements (a), distributed model results against measurements (b), and distributed model results against NARX results (c). The regression line, its equation, and the CC are specified on the graphs, along with the envelope at ±15% represented by the pink dashed lines.
As mentioned earlier, local authorities and regulatory services prioritize overflow volume as the primary operational criterion. This volume is an integration of the discharged flow over a period (usually a daily aggregation). Figure 7 shows a pairwise comparison of simulation results and measurements of daily overflow volumes for the year 2018. The three comparisons showed strictly equivalent results in terms of CCs. The detailed analysis of events with low daily volumes revealed that NARX tended to generate small overflow volumes more frequently than what was actually measured. This tendency was less pronounced for the distributed model. Conversely, some daily overflow volumes were significantly overestimated by the distributed model, corresponding to values between 5,000 and 7,000 m3/day (Figure 7(b)). This deviation was also evident in the model comparison graph (Figure 7(c)), as this overestimation was not replicated by NARX. It seems that the most likely explanation for these deviations lies in the quality of the measurements, probably the description of the rainfall field. The CSO events concerned occurred between the end of May and the end of August, a period when localized storm events are common, and the heterogeneity of the rainfall field is most pronounced. It is likely that the use of the three rain gauges for the development of the distributed model did not adequately account for the actual distribution of rainfall. Finally, it is worthwhile to note the very good correlation between overflows simulated by both models for daily volumes exceeding 10,000 m3/day. The correspondence was even excellent for volumes exceeding 20,000 m3/day.
The comparative analysis of results produced by the two models within a validation framework leads to the conclusion of their good performance for daily overflow simulation as well as for hydrograph simulations.
Model performance and time required to build and run models
As demonstrated earlier, both types of models considered in this study performed well enough to be used as simulators for predictive or projection purposes of sewer overflows. However, it is essential to keep in mind that the construction of the neural network in the present case is an exercise that may be perceived as lacking robustness. Indeed, the systematic construction of 30 networks for each tested configuration resulted in some networks with very low performance. These performance variations can be explained, firstly, by the fact that for each repetition, the dataset was randomly divided into three parts (training, validation, and testing) – differently for each repetition. Secondly, the performance variations can be due to the random nature of the initialization of neuron weights at the beginning of the optimization process. The applied method involved selecting, among the 30 available networks for the same configuration, the one with the best performance. In this study, the choice of the number of repetitions for each configuration was primarily motivated by the need to control computation time. The search for the optimal number of repetitions to achieve the best possible network was not addressed. An optimization of the number of networks to be tested to select the suitable candidate in terms of performance could be the subject of future research.
From an operational point of view, a criterion typically considered due to its impact on the study budget is the time required for model construction and adjustment. Unfortunately, it was challenging to make a definitive statement on this matter because the distributed model developed in this study used an existing software, and the datasets were already existing (without additional measurement campaigns). However, we could empirically consider that the order of the time ratio needed to refine a neural network versus a distributed model would be around one to 10, assuming an initial situation without any pre-existing model and an equivalent level of expertise of the modeller in both modelling techniques. This would not seem to be exaggerated from a strictly computational time perspective when compared with the literature. Ráduly et al. (2007) compared a conceptual model of a wastewater treatment plant with a neural network and highlighted a reduction in computation time by a factor of 36 in favour of the neural network when the learning phase was included in the calculation time. This factor increased to 1,300 when using a pre-trained network. Because the studied object was different, these ratios are not directly transposable to wastewater networks but indicate a comparison between methods. Concerning the computation time in our case, it is worth mentioning that the performance of the neural network is superior to that of the distributed model. Whereas simulating an entire year required approximately 12 min of computation for the distributed model, the neural network provided results in around 2 s.
Beyond the efforts required to build the models and the differences in computation time, each of the two types of models is not suitable for all uses. Neural networks find applications when real-time management of networks is at stake, and the calculation duration parameter is crucial. Distributed models are more commonly used for diagnostic studies and analysis of development scenarios.
Effect of the quantity and the reliability of measurement dataset
Another point worth mentioning is the quantity and reliability of data used to train the neural network. A period of recorded overflows includes both discharge and non-discharge periods. In 2019 and the first half of 2020, only 1.3% of overflow values were non-zero. Even though simulating the absence of discharge is an important feature sought in the model's operation, the 1,383 time steps with non-zero values recorded in 2019 and the first half of 2020 make up a relatively small dataset. Despite that, the dataset used seemed sufficient to build and adjust an ANN model. Indeed, the neural network was effective according to standard performance criteria. The approach developed here aimed to determine if a NARX-type model can be a valid alternative to a distributed model for conducting rainfall/overflow simulations. Therefore, it was necessary to adopt an identical configuration to build both models. The method chosen to construct the NARX utilizes the available data in the same way as for the construction of the distributed model. Furthermore, the operational aim of this work is to seek conditions that may be encountered in a study context. The quantity of data at our disposal, namely the two and a half years of data used to calibrate and verify the models, aligns well with the operational context, where long histories of validated data are rarely available.
CONCLUSION
The objective of this paper was to determine whether a neural network can be an acceptable alternative to a distributed model for simulating overflow discharges through a CSO. We demonstrated that a NARX neural network with a configuration of 10 neurons in the hidden layer and a delay of six timesteps achieves acceptable performance regarding common hydrological performance evaluation criteria. Under identical construction and performance evaluation conditions, we could conclude that the two models performed similarly. The neural network shows real operational advantages due to its ease of implementation and is therefore well-suited for simulating the impact of long-term rainfall series on overflows as long as the sewer network remains unchanged. On the other hand, a distributed model becomes essential in case of modifications to the structure or operation of the sewer network, when volumes and frequency of overflows may change. The superiority of one model over the other in terms of extrapolation capability – that is, the performance in producing output when the input is outside the range of inputs used to calibrate and validate the model – remains unknown. It is indeed challenging to determine because, by construction, if extrapolation is considered, it implies dealing with unprecedented situations for which there is no available data. The issue of extrapolation is addressed in the literature on empirical models, including specifically in hydrology topics (Hettiarachchi et al. 2005). However, it is less explored when it comes to distributed conceptual models, even though the extrapolation capabilities of a distributed conceptual model are also not always well known.
ACKNOWLEDGEMENTS
The authors extend their gratitude to the teams from Valence city for granting access to all data relating to the structure and operation of the sanitation system. This study was conducted in the framework of OTHU (Observatoire de Terrain en Hydrologie Urbaine) research federation and EUR H2O'Lyon (ANR-17-EURE-0018).
DATA AVAILABILITY STATEMENT
Data cannot be made publicly available; readers should contact the corresponding author for details.
CONFLICT OF INTEREST
The authors declare there is no conflict.
%20cropped.png?versionId=7142)