Sewer sediment deposition prediction using a two-stage machine learning solution

Combined sewer systems (CSSs) merge rainwater runoff, domestic sewage, and industrial wastewater in the same pipeline. Most systems transmit the combined sewage to wastewater treatment plants for treatment prior to its discharge into a water body. During the transport of wastewater, arid and solid sediment, fat, oil, and grease accumulate in some parts of the network. This event reduces the volume of water that can be carried through the pipe, and sometimes the build-up can be so large that failures or blockages can occur (Ashley et al. 2000, 2005). These events pose a high environmental and social risk, as they can cause retentions or flooding in the city with water containing toxic materials and substances. In addition, the sulphide derived from the retention of the wastewater will generate odour problems affecting the population and accelerate the sewage system's degradation and corrosion. Preventive routines involve a significant investment in terms of financial resources and human labour, but they are essential for maintaining the proper flow of wastewater through the sewer system by continuously monitoring its status and removing sediments as needed. For instance, the cost of maintaining and expanding the piping network in the city of Munich from 2000 to 2021 is estimated at €192.3 million (Al-Azzawi et al. 2022), and the US Environmental Protection Agency (EPA) estimated in 2016 that $271 billion were needed to maintain and improve the national wastewater infrastructure. A portion of this investment, approximately $48 billion, is allocated for correcting Combined Sewer Overflow (CSO) issues, which occur during wet weather conditions and result in the discharge of untreated wastewater and stormwater (Daguillard 2016).

Nowadays, water organizations extract and save the information generated during maintenance routines, resulting in a valuable pool of exploitable data. This information can be used to train data-driven models and develop predictive maintenance (PdM) solutions. The concept behind PdM is to use advanced analytical tools to anticipate when maintenance or repairs are required. Organizations can analyse patterns and trends in the data obtained during historical maintenance routines to predict when the infrastructure is likely to fail or require maintenance. This technique enables cost savings by allowing for the postponement or reduction of certain routines while also improving the allocation of resources dedicated to specific tasks (Kumar et al. 2019).

A diverse range of machine learning (ML) algorithms can be used to implement PdM solutions (Carvalho et al. 2019; Ran et al. 2019; Çınar et al. 2020). Determining the state of objects or infrastructures in the short or long term can be achieved using supervised or unsupervised classification algorithms. Similarly, supervised regressive algorithms can be used to predict the remaining useful life or degradation of an object or infrastructure over time.

Studies show several ML techniques applied for sewer system condition prediction, where some of the main focuses are blockage, failure, and sediment predictions. Bailey et al. (2015) show the use of static and dynamic sewer data to create a predictive model based on decision trees that calculates the blockage likelihood of a sewer section. In a later study (Bailey et al. 2016), they show the addition of geographical grouping of sewer sections to improve their research. Hassouna et al. (2019) introduce the use of maintenance duration and sewer level monitoring data to train ensemble algorithms like a random forest to predict blockage incidents, and Okwori et al. (2021) explain the use of spatial heterogeneity to predict blockage propensity in sewer sections using a Poisson regression, and the prediction of blockage recurrence using random forest.

In the use case of failure prediction, the data and algorithms used to generate the predictive models are similar. Tavakoli et al. (2020) generate a random forest using data from the mainline sewer pipes, following a methodology like the ones already mentioned. Other types of algorithms are also being researched nowadays. Sattar et al. (2019) compare artificial neural networks (ANNs), a support vector machine (SVM), and a non-linear regression using a dataset containing 80 years of recorded failures within a sewer network. Genetic algorithms are also used in the domain to optimize the hyperparameters of predictive models during the training process (Robles-Velasco et al. 2021). Even in studies where the use cases are based on predicting suspended sediment load on rivers, the range of algorithms tested varies from traditional ML such as Random Forest (RF) or SVM, to deep learning algorithms such as ANN and specific architectures such as long short-term memory (Allawi et al. 2023a, 2023b).

Sediment problems are usually solved by focusing on sediment transport amount or deposition level as an objective variable. Song et al. (2018) build a predictive model based on numerical analysis using a two-phase calculation of fluid flow analysis and sediment accumulation, using data of non-dimensional variables for inlet velocity, inlet particle volume fraction, and particle size. Mohtar et al. (2018) compare ANN and radial basis function models to obtain the best predictions over the incipient sediment motion in sewers using the parameters: varying sediment thickness, median grain size, and water depth, demonstrating the potential of using ANN as it is the best model in their study. The potential of ANN is also demonstrated for sediment deposition in CSSs when compared to more traditional ML algorithms (Ribalta et al. 2021). Finally, feature reduction is also a relevant topic within sewer predictions. Bhagat et al. (2021) propose a solution to predict sediment lead using a two-phase solution that involves a first step of variable selection using extreme gradient boosting and a second step using another ML algorithm such as ANN or SVM.

Our study stems from previous work (Ribalta et al. 2021), where we compare predictive models to classify the sediment deposition percentage level in the present and ten days ahead, and another approach that compares models to classify if a sewer section needs maintenance. The novelty in this previous study is the use of the information provided by similar neighbour sections to improve the data being fed into the algorithms and the produced predictions. However, the study has a problem with predicting sediment deposition. The obtained result pertains to present deposition within a section, which may not be as useful for water utilities since they need to schedule maintenance days in advance, and the prediction assumes that the maintenance needs to be done at that moment. Another problem arises from the data of nearby sections since utilities must access these sections to extract current deposition data, and in many cases, maintenance has already been applied to the section to be predicted. Predicting future sediment deposition increases planning time and opportunities for utilities.

In this new work, our goal is to enhance the accuracy of predicting future sediment deposition and approximate the amount that will occur before the next maintenance, which could take place in the next 3–12 months. However, this research poses a significant challenge due to the dataset's complexity, which comprises only a limited number of historical records but a high number of features – 85 features spread across 1905 records. Consequently, the dataset's structure creates uncertainty for the model, making it arduous to establish correlations between the features when there are few records available for training.

We aim to overcome this challenge by proposing a novel ML-based approach that employs a two-phase solution with a categorical objective variable, unlike the traditional continuous variable. The first model we present compresses information from 49 physical variables into 15 output variables, while the second model utilizes this output information together with 36 dynamic variables to classify sediment deposition status. To validate our models, we use data obtained from historical records of three neighbourhoods in the CSS of Barcelona.

Our proposed data model leverages not only the data from individual sewer pipes but also features from pipes that exhibit similar deposition behaviour. With this approach, we are confident that our models will deliver relevant scores in predicting future sediment deposition while using fewer features, thereby eliminating information overload and providing a more efficient and practical solution.

The dataset we analysed provides network information on three neighbourhoods in the City of Barcelona, covering the period between 2017 and 2021. Table 1 presents the historical dataset's variables, including physical features, water and climate behaviour indicators, and maintenance registers for each pipe, bed, sections, or other types of structures (we use the word pipe or section to address them within the study), also known as dynamical features in this study. Notably, the dataset comprises diverse records from various sewer sections, ranging from small diameter pipes of 0.15 m to massive ones exceeding 2 m in diameter. The pipe length is also highly variable, with crosscutting pipes measuring a mere 0.5 m and main waterways extending up to 74 m. Maintenance routines are not performed on a scheduled basis but are instead planned based on urban management plans, sewer system risk prevention, and other factors, causing inspection dates to fluctuate. However, one of the most critical aspects of the dataset is the interval between maintenance routines, which can range from 2 months to over a year. This variation amplifies the uncertainty of determining the deposition level's evolution since the last inspection. Rainfall also has a significant double impact on sediment deposition, increasing the wastewater flow and, as a result, the risk of blockage or natural flushing. Therefore, the daily rainfall data collected by three rain gauges, one for each neighbourhood, provides a crucial indicator of trend changes in the deposition level, making it an essential component of our dataset.

Our data model augments each record, as presented in the previous section, appending two types of information: the state of the pipe at previous maintenance sessions, and physical and dynamic features from sections near the pipe. The result of analysing and processing the data can be seen in Table 2. The physical features are the same as described in Table 1, and most dynamic features are modified. The biggest change between the dataset and the data model is the use of data from near and similar sections. We incorporate data from similar sections to improve accuracy, selecting the five sections with the most similar sediment deposition trends and including their features in the data model. The similar (near) sections are all in the same neighbourhood, so the feature indicating it is only present for the section to be evaluated. Additionally, we enhance sediment level and cleaning features by including past values to understand better the accumulation trends. Using the sediment level in the historical data, expressed in centimetres, and comparing it to the height of the section, we compute the proportion of the section that the accumulated sediment occupies, represented as a percentage. We find that discarding the days between maintenance routines leads to better results, so we remove this feature. Lastly, we aggregate daily rainfall data between maintenance routines into two variables: the average rainfall and the standard deviation. The former provides an overview of the rain amount during the period, while the latter considers all days with intense rainfall. Therefore, the days between maintenance routines are also represented in the rainfall impact on the sewer section, and the sediment accumulation trends for that same section let the model assume the possible increment between maintenance sessions. Overall, the total number of registers amounts to 1905.

The available data are divided into two types of features: constant values of a pipe called ‘physical’ and variable historical values called ‘dynamic’. However, with a final count of 85 features, predicting sediment accumulation using these features can be challenging. An ML algorithm requires many records to understand the existing variables. Thus, increasing the number of variables in the dataset necessitates an increase in records (Sorzano et al. 2014). Fortunately, the physical features, composed of constant values, have lower overall variance within the dataset and are more redundant than the dynamic features. Therefore, we propose a methodology to pre-process the physical features using an autoencoder (AE) architecture to compress them to a smaller number of features without significant information loss. We then join these compressed physical features with the dynamic features to predict sediment accumulation using an ANN.

An AE (Kramer 1993) is a type of ANN used to learn the relation between the different features of a dataset. It consists of an encoder, which compresses the information into a lower space, and a decoder, which tries to decompress the information into its original state. It has diverse applications, such as image denoising, image reconstruction, or dimensionality reduction (Kramer 1993). AEs shine in unsupervised learning scenarios, leveraging the inherent structure of the data to learn meaningful representations without the need for labelled datasets. By encoding data into a compressed latent space and then decoding it back to its original form, they facilitate the discovery of fundamental data characteristics, creating a more concise and meaningful feature representation. This transformation is particularly beneficial for complex datasets, where the reduction of dimensionality can significantly improve the performance of subsequent predictive models. Although other methods of dimension reduction have not been tested in this study, the authors have explored other options and decided on AEs due to their theoretical and practical advantages. For instance, unlike linear methods such as principal component analysis, AEs can capture non-linear relationships in the data, which is crucial for complex datasets where linear assumptions do not hold (Wang et al. 2016). AEs are capable of learning features automatically from the data, eliminating the need for manual feature engineering – a significant advantage in high-dimensional spaces where manual feature selection is impractical. Additionally, one of the main ideas is to study an ML solution that can be reused or re-implemented with ease in other case studies. Other dimension reduction methods require meticulous analysis and expert knowledge to decide the number of features to reduce, but to decide the number of features being reduced by an AE, the methodology is based on the scores and performance of the predictive model. In this study, we apply the architecture to compress the information of the physical features during the encoding phase and the result is sent to the ANN predictive model, specifically a feedforward neural network with a multi-layer perceptron. The decision to use the ANN comes mainly from the conclusions drawn in the previous study, where it is shown that the feedforward neural network trains better with our data to make future predictions (Ribalta et al. 2021). Nevertheless, ANNs are versatile and powerful models capable of capturing complex non-linear relationships in data, making them suitable for prediction tasks where the relationship between input features and the target variable is not linear or easily discernible.

The predictive model uses the 36 dynamic features shown in Table 2 and the compressed physical features to identify the future percentage of sediment level accumulated within a pipe. Concretely, the aim is to classify the percentage of sediment accumulation occupying the pipe, divided into five different classes: 0–5%, 5–10%, 10–15%, 15–20%, and 20–100% deposition level. The number of registers indicating an accumulation of more than 20% comprises 3% of the total number of registers available within the dataset, and since any value higher than 20% is dangerous for the water utility, we decide to classify them using the same class. The classification of these different classes, each made by a different model, can be used together to obtain a more specific indicator of the specific range in which the deposited sediment can be found. For instance, when one model indicates that sediment deposition exceeds 5% (True) and another model suggests that it does not surpass 10% (False), we can deduce that the sediment deposition falls within the 5–10% range.

The prediction of sediment deposition is usually done with regression algorithms, but in our case, a classification problem offers several advantages. First, it facilitates handling the dataset's sparsity and the high number of features by focusing on categorical outcomes rather than exact numerical predictions, which might be highly variable or uncertain due to the dataset's constraints. Second, a classification model provides actionable insights into water utilities by categorizing sediment deposition levels into predefined classes that directly inform maintenance scheduling decisions, enhancing operational efficiency and planning.

To generate the two-step ML solution, we need to train and evaluate the architectures in a specific order. Table 3 shows the different steps carried out during the study to train the ML architecture. We initially define, train, and evaluate several AE architectures to compare and decide which are suitable for the data model. The decision is based on the performance of a single perceptron, which receives the output results of the AE and uses them to predict sediment deposition. Afterwards, the same methodology is applied to define the ANN architecture. Those are trained using the data model without applying any encoding conversion, so the results are not biased by any AE pre-processing. The final step is to build different combinations from the best AE and ANN architectures, train and evaluate the predictions, and select the most scoring one.

Neural networks have multiple parameters to configure, and the number of possible combinations is high. Training all possible neural network architectures would take a lot of time and resources, so the best workaround is to previously filter certain configurations and train only the ones with the best expectations to perform. In the article, we show a subset of architectures that obtain better performance and indicate patterns between architectures, such as the best number of layers or the range of neurons with better scoring.

To determine the most effective predictive architectures, we utilize various metrics that allow us to compare trained models, identify predictive weaknesses, and evaluate overall algorithmic performance. We employ six distinct metrics to evaluate each model, as outlined in Table 4, with each metric providing a unique perspective on the performance of the algorithm:

• Accuracy: Measures the proportion of correct classifications of sediment deposition among the total dataset.

• Specificity: Measures the proportion of real sediment deposition with less value than the threshold classified correctly by the model.

• Sensitivity: Measures the proportion of real sediment deposition with more value than the threshold classified correctly by the model.

• Precision: Measures the proportion of predicted sediment deposition with more value than the threshold classified correctly by the model.

• Area under the curve (AUC): The receiver operating characteristic (ROC) curve plots the true positive (TP) rate and the false positive (FP) rate at different classification thresholds, producing different bidimensional points ranging from (0, 0) to (1, 1) with the shape of a curve. The AUC is the entire area underneath the ROC curve, and it provides an aggregated performance of how well the model performs at various threshold settings (Bradley 1997).

• Average specificity-sensitivity (ASS): Measures the balance between the two metrics from a general perspective.

Table 4

Evaluation metrics

Statistical metrics .	Formulas .	Optimum score .
Accuracy		1
Specificity		1
Sensitivity		1
Precision		1
AUC	The area of the ROC curve.	1
ASS		1

Statistical metrics .	Formulas .	Optimum score .
Accuracy		1
Specificity		1
Sensitivity		1
Precision		1
AUC	The area of the ROC curve.	1
ASS		1

Note: A positive is a register with more deposition than the threshold and a negative is a register with less deposition than the threshold.

View Large

The evaluation of the drafted AE architectures is shown in Table 5. The first row details the results of the predictive algorithm without using an AE to show the benefits of using the AE in a basic use case. During the first 50 epochs of training the neuron, not using an AE provides the best sensitivity out of all options. The ASS is 0.74, being the best value in epoch 50 but not the best of the three evaluated epochs, being surpassed by other architectures.

Regarding the number of layers used, the maximum specificity obtained is 0.71 when using two layers to encode the data, 0.72 when using three encoding layers, and the maximum sensitivity is 0.83 and 0.84, respectively. Table 5 displays combinations with different activation functions, showing the performance of the architectures using a relu or sigmoid activation. Overall, both perform well, but the highest performance is accomplished using the sigmoid activation function, with a maximum specificity of 0.71 and a sensitivity of 0.84. The architecture with the best specificity has 30-20-10-20-30 neurons per layer and sigmoid activation function, with 0.72 specificities at epoch 50, and the best sensitivity is offered by three different architectures scoring 0.84 at epoch 100.

To decide which architecture performs better overall, the ASS is used, selecting the highest. The results indicate two architectures with the same score: the 5-layer AE with 30-20-15-20-30 neurons per layer using the sigmoid activation function and the 5-layer AE with 40-30-10-30-40 neurons per layer using the relu function, both with an average score of 0.75.

Table 6 presents the best subset of ANN architectures, trained without using an AE and using different layer sizes and activation functions. The architectures have a common pattern of three layers each and a sigmoid as the final activation function. The best specificity is offered by the 20-20-1 and 10-10-1 neural networks, both using the activation function softsign, with a score of 0.78, and the best sensitivity is offered by the 20-20-1 neural network using a logarithmic sigmoid, 0.84.

Two architectures offer the best score, the 10-10-1 and 20-20-1 neural networks using a softsign activation function, both with an ASS of 0.8 at epoch 50. However, the 20-20-1 neural network has the same score at epoch 100, being the decisive factor to select it as the best neural network.

The final step is to determine which combination of AE and ANN offers the best results. Table 7 shows the scores of the top two AE combined with the best ANNs. The ANN is trained during 50 epochs, uses a softsign activation function, and uses a mean squared error loss, but each AE is trained using various settings to maximize the final score. Table 7(a) displays the results of training a 5 neurons per layer ANN, and the AE with a 40-30-10_Relu architecture outputs a 0.8 ASS score, improving the ASS by 0.02 from training without an AE. Table 7(b) shows the results for a 10 neurons per layer ANN, but in the case of using an AE to boost the train outputs the same ASS, 0.8. Table 7(c) presents the results for a 15 neurons per layer ANN, and the 40-30-10_Relu scores the best ASS, 0.82, which is a 0.02 increase from training the ANN without AE. Lastly, Table 7(d) displays the result of using an AE to boost the training of the 20 neurons per layer ANN. The best ASS is achieved by the 30-20-15_Sigmoid AE, increasing from 0.81 to 0.82.

We extend the analysis beyond the integration of AEs and ANNs by comparing their performance against traditional ML techniques, specifically focusing on how these methods fare when utilizing the outputs generated by the AE. The models selected for this comparative analysis were previously employed in an earlier study (Ribalta et al. 2021). However, due to the evolution of our data model, which now incorporates additional variables such as rainfall, a retraining process was necessary to ensure their current applicability. To identify the optimal configuration for each model, we conducted a grid search. This process involved experimenting with various parameter combinations to determine which settings yield the most favourable outcomes. The output of this process produced the following models and configurations:

• Gradient Boosting: 200 estimators, maximum depth of 3, and learning rate of 0.1.

• AdaBoost: 200 estimators and learning rate of 0.1.

• RF: 200 estimators, without restricting the maximum depth.

• Extra Trees: 200 estimators and maximum depth of 20.

• Logistic Regression: Inverse of regularization strength, C, of value 10.

To train and evaluate these models, we used the same methodology that has been used with the neural networks, and the results are shown in Table 8. The models have higher scores in specificity than the neural networks, but in the case of sensitivity and the ASS metrics, the scores are lower. Of the models shown, the one with the best scores is the Gradient Boosting, with a final ASS value of 0.74, being 0.08 lower than the best ANN configuration. Therefore, the ANN provides better results than any of the traditional algorithms presented.

To ensure the validity of the proposed architecture across different classification thresholds, Table 9 presents the evaluation of four independent models using various scoring metrics. An important factor to consider when assessing the results is the balance of positive values in the dataset, or, said differently, the percentage of registers that are higher than the evaluated threshold. The dataset with the lowest threshold (>5%) exhibits a balance of 39%, which decreases as the threshold increases, to 14, 4.82, and 3.25% for the thresholds >10%, >15%, and >20%, respectively. This imbalance significantly impacts on precision and sensitivity scores, as they tend to decrease in value due to the high penalty incurred by the low number of real positives to be predicted compared to the high likelihood of producing FPs. Conversely, accuracy and specificity benefit from the imbalance due to the increase in true negative (TN) values and the model's ability to correctly predict a negative value.

When examining the real sediment depositions that exceed 15 and 20%, it is interesting to note that our models never predict them as having no deposition, and instead, mostly predict a deposition of over 10%. The significance of this observation lies in the models' ability to distinguish between no deposition and significant deposition events. Similarly, for increases greater than 10%, only a handful of cases were predicted as having no deposition. As the actual deposition values decrease towards the threshold for no deposition, we see an increase in false predictions, with over 100 instances of FNs at the 5% threshold. Looking at FPs, we observe that nearly 400 records were wrongly predicted as having a sediment deposition, with many instances of >5% predicted as higher thresholds. Particularly, the >15% threshold suffered the most, with only 9 correct predictions. It is worth evaluating the extent of erroneous predictions. In the case of >20%, the worst false prediction would be predicting no deposition, and the best would be predicting >15%. False predictions are often adjacent to the threshold being predicted within the confusion matrix. The >15% threshold produced the best results, whereas false predictions were typically in >10% and >20%.

The <5% threshold holds most values within the correct range, with a maximum value of 7 and a minimum value of 0, and some higher outlier values. The >5% threshold has most values between 3 and 8, with a median value of 5.5, a maximum value of 15.5, and a minimum of 0. The >10% has most values between 4 and 11, which matches with the 226 under threshold classifications indicated in Figure 4, and has a maximum and minimum value of 18 and 1. The >15% has most values between 6 and 17, with a median value of 12, a maximum of 33, and a minimum of 3. Finally, the >20% encompasses a longer range of values, between 5 and 20, with a maximum of 40 and a minimum of 1.

When comparing the various thresholds, it becomes clear that the <5% threshold can correctly classify the range of values, but the other thresholds tend to include some positive values that are falsely classified within the 0–5% range. The >5% and >10% thresholds tend to have most values falling within a similar space, but the maximum and minimum values classified differ, with the latter having a higher range. Moving on to the >15% and >20% thresholds, these two have a much wider range of values, but the >20% threshold seems to suffer from a large number of FPs due to the decision-making process defined earlier. Overall, the thresholds tend to show an increasing deposition value range, with higher deposition values being classified in higher thresholds.

Sediment deposition estimation in a sewer section is a challenge that has been researched for a long time (Ribalta et al. 2022), studying predictions in real time, short term, and long term. However, attempting to build a predictive solution without sensor data along the entire sewer requires a dataset that represents the behaviour of the sewer at a general scope and further optimization of the features to extract all possible details. The complexity of the problem lies in the structure of the information since generating a descriptive data model requires enough features to indicate the physical properties of the infrastructure, the applied maintenance, and the environmental effects that can influence the accumulation process. A high number of features need to be followed by a high number of registers to enable enough learning information for the algorithms to comprehend the inner behaviour of each parameter and the generic relationship between all of them and the objective variable. This fact directly impacts the type of algorithm to be used to train since many state-of-the-art architectures require many records to train correctly (Sevilla et al. 2022). Furthermore, the number of articles on the research topic yearly is not high (Ribalta et al. 2022), and there is no opportunity to compare a wide range of algorithms applied on multiple occasions within the domain of sewer sediment accumulation. While past studies use data-driven models to predict sediment deposition at the present (Jiang et al. 2021; Ribalta et al. 2022; Vanegas et al. 2022), the proposed model predicts sediment deposition in the future maintenance session. Additionally, as demonstrated in a study by Rosin et al. (2022), incorporating rainfall information into the predictive model is beneficial and might be necessary for future studies of this type.

In this study, we use AEs to reduce the high number of input features and refine the final prediction, but its benefits to the predictive model are not big. The data of this case study have an imbalance between the number of features and the total number of registers, so obtaining more maintenance registers of the studied sections would benefit the training process of the predictive model. Therefore, there is still room to improve and demonstrate the AEs' performance when the number of registers is consistent, not being affected by the curse of dimensionality.

Predicting future sediment deposition in this study is subject to certain challenges. The ideal scenario would involve an exact and regular time interval between maintenance sessions, but this approach is being replaced by the dynamic optimization of maintenance schedules (Tscheikner-Gratl et al. 2019). In most sewer sections in this study, the interval between maintenance sessions varies, making it difficult for the model to interpret and understand seasonal variations in sediment deposition. Nonetheless, this study proposes an architecture capable of predicting a range of future deposition within various time horizons, which can be replicated in other case studies. The objective variable is an essential component of the predictive solution, providing a small range of deposition values that can be effectively utilized by water utilities to aid in their maintenance sessions, despite not offering the exact deposition value.

The predictive ranges presented in the study are selected from the historical maintenance data, considering the number of registers available for each range and category. For this study, the range with more registers to offer is the <5% threshold, with 1,156 registers, and is also the one with the best predictions. As the threshold increases, the dataset becomes increasingly unbalanced, with fewer registers above the threshold, leading to a decrease in the quality of predictions, as it happens in many other ML problems (Thabtah et al. 2020; Ghosh et al. 2022). Nevertheless, the model predictions keep providing reliable information to the water utility since the percentage of TNs is higher in most cases.

Using the trained models in the four possible thresholds gives more information as a final output, but the results are also more difficult to understand. We provide a simple interpretation of the results, prioritizing the predictions with a higher sediment deposition, although there are other potential ways to improve this process. When the models predict, the output value ranges between 0 and 1, indicating the confidence value of the model towards predicting if the deposition will meet the threshold or not. In this case, a final function could be defined to decide which of the models' outputs has sufficient confidence to make the most likely prediction. This function could be defined using a new data-driven model, receiving the output of the models and deciding which of the thresholds is correct.

Water utilities can leverage the models as decision-support tools to plan future maintenance sessions for each section. The results indicate several conditions met by the models that can be used in some scenarios. When using all the models, the classifications into the class <5% are mostly correct, providing a clear indication of the sections with minimal deposition. While utilizing each model individually, one must be aware of the implications. The >5% model can be employed in all situations due to the stability in the predictions and good behaviour, but the other models produce a high number of FPs, rendering them useful only when applying maintenance to sections with a lower expected deposition that has no adverse impact. Additionally, the models can accurately classify situations where the result should be positive, with a low number of FNs.

In this study, we show the application of an ML architecture to predict the future sediment deposition in sections of a sewer system. Using real data from the city of Barcelona, we present an architecture that handles a dataset with the curse of dimensionality, objective variable imbalance, and long-term estimations.

In addressing the curse of dimensionality, our architecture incorporates a data reduction technique through the use of an AE, which adeptly compresses variables that represent the static physical properties of sewer sections. This approach streamlines the dataset and enables the predictive model to more effectively discern the relationship between predictive and objective variables with less historical data. Furthermore, the AE output is used by an ANN, but there are more algorithms and architectures to be tested in other case studies since feature reduction may be beneficial to algorithms that require a low number of variables. The current architecture is being tested using a rather low number of historical registers and doing it with a high number would let the community know if the impact on data reduction positively increases the deposition classification. We also compare the performance of the architecture with traditional ML algorithms within the study. Combining the AE and the ANN, we obtain a score of a specificity of 0.76, a sensitivity of 0.88, and an average of 0.82 between them (ASS), and the combination of AE with the best-performing algorithm (Gradient Boosting) results in a score of a specificity of 0.82, a sensitivity of 0.66, and an average (ASS) of 0.74, meaning that within our study, the ANN performs better.

These tests have been carried out using the limit that considers sediment deposition greater than 5%, but the chosen architecture is evaluated in four different classification models, ranging from 5 to 20% sediment deposition. This method reduces the complexity of estimating the future sediment deposition value by using a category that englobes a range of possible values instead of focusing on predicting the exact value, enabling long-term predictions of sediment deposition. In our study, we use four ranges, but future studies might have a different data structure with a different balance of positive values that may enable the study of higher thresholds. Those cases would need to redefine the categorical ranges by looking into the balance of positive and negative registers and verifying that the presented architecture works properly using our scoring methodology.

Although the study presents solutions to prevalent issues within sewer system datasets and outlines strategies for long-term prediction, there is a need for further exploration into several areas. Among these is to enhance the dataset diversity. Demonstrating the architecture's efficacy across various deposition scenarios necessitates increasing the volume of data, particularly data reflecting higher sediment deposition percentages. Expanding our dataset would facilitate the examination of different thresholds and identify which yields the most insightful information and which can be most accurately predicted using this architecture. It is important to emphasize that the objective of this study is not to simulate the actual behaviour of the sewer system with the predictions but rather to enhance sewer management decision-making. Therefore, the necessary data should not be related to real-time sediment behaviour, such as sediment transport.

To enhance the effectiveness of future studies in this field, it is crucial to identify the relative importance of different features in the model prediction process. This would boost the data preparation process, allowing more time to be devoted to other crucial steps in model development. In addition, a thorough understanding of feature importance can reduce the time spent to decide which data need to be collected, thereby optimizing the overall research process. An important technique that could be used in future studies to identify this importance is explainable artificial intelligence, which analyses the impact of the different features in the predictions done by the ML models.

The work described in this paper has been conducted within the project SCOREwater. This project has received funding from the European Union's Horizon 2020 Research and Innovation Programme under grant agreement no 820751. Marc Ribalta acknowledges funding from AGAUR DI-2019-066.

Data cannot be made publicly available; readers should contact the corresponding author for details.

The authors declare there is no conflict.