Data-driven approaches for pipe life prognosis in water distribution networks: the Barcelona water distribution network case study Open Access

Water scarcity problems are due to climate change because of increasing demand and important loss during transportation in many cities in the world. Water distribution networks (WDNs) are critical infrastructures dealing with water supply and distribution to customers which are growing in complexity, and simultaneously facing aging pipe failures with time. According to the American Environmental Protection Agency (EPA), around 30% of water pipes need immediate action (Allbee 2005). Maintaining or renewing these aging pipes requires a large amount of investment. In the member states of the Organization for Economic Co-operation and Development, around 0.5% of the annual gross domestic product is needed (OECD 2006). Furthermore, according to the water balance report from the International Water Association (IWA), pipe failures can lead to up to 27% of the total extracted water loss (Berg 2015). Moreover, maintaining or renewing aging pipes faces a number of barriers, due to the lack of adequate knowledge to assess the assets state, particularly the ones in the underground. Innovative tools, mathematical models and geographical information systems assess pipes’ health and predict their remaining useful life (RUL) in a WDN can reduce the investment of asset.

However, it is quite complex to assess the pipes’ life due to the knowledge required for the influencing factors and their relations with their failure . There are mainly three types of approaches to assessing pipes’ life (Winkler et al. 2018): reliability, physical degradation, and data-driven models. Among these, the reliability approaches characterize the failure probability of the buried water pipes, using historical data, which allow one to determine their RUL (Pietrucha-Urbanik & Pociask 2016), depending on important parameterized factors, such as diameter, material, and age (Su et al. 1987; Fujiwara & Tung 1991; Khomsi et al. 1996).

Beyond reliability, the degradation model uses physical principles to describe pipes' aging process. The event type, the smallest element to be described, and the process to be modeled are the three dimensions that classify degradation models (Kleiner & Rajani 2001). Event type can affect and characterize the models of failure occurrence and RUL by representing different processes of physical degradation. The dimension of the smallest element decides whether the study object is a large or a small scale of a pipe network. All three dimensions can be integrated for specific objectives.

The data-driven approach was reviewed by Scheidegger et al. (2015). Among these, the dependance of the failure rate on different features (diameter, age, and total length) has been characterized using evolutionary regression models (Berardi et al. 2008). Moreover, a neural network (NN)-based approach, the adaptive neuro-fuzzy inference system has also been proposed (Tabesh et al. 2009), to map the pipe failure rate with pipe pressure, installation depth, length, age, and diameter. Compared with other methods such as different non-linear regression, and reliability models, NN-based models work much better. An automatic damage segmentation framework for buried sewer pipes, based on machine vision techniques using a dataset of 3,558 images, was proposed by Wang et al. (2023). However, the images of buried pipes are quite difficult to obtain, especially to analyze the historical recordings 100 years ago. The performance of different machine learning models, for pipe breaks using datasets from multiple utilities, was evaluated by Chen et al. (2022), concluding that reliable historical break data affect significantly model accuracy. Machine learning was combined (Snider & McBean 2021) with survival statistics to predict the remaining service life of water mains. However, although various research studies have covered enough of the occurrence law and association of pipe failures with different features (Sun et al. 2020), an applicable approach that can assist water managers in their daily maintenance and management of their water assets is still missing.

Due to the fast development of sensing and communication techniques, a vast amount of data are available, which lead to groundbreaking advances in data-driven approaches. While turning to pipes’ life prognosis, a time-to-event prediction problem can be considered, which can predict when a future event will happen and why. For the time-to-event prediction problem, Kvamm et al. (2019) proposed one method that integrates the survival model with machine learning by extending the Cox proportional hazards model with neural networks (CoxMLP). In this way, the flexibility of neural networks can be added while modeling events using continuous time, which could be a good approach for pipes’ life prognosis. Besides survival models based on CoxMLP, evolutionary polynomial regression (EPR) is also a novel method that can provide optimal solutions by processing and learning input parameters with a large amount of data (Kvamm et al. 2019). Due to its strong capacity to handle noisy data with missing input, the EPR approach was first applied to the water field (Giustolisi & Savic 2006), and then has been widely used in predicting river discharge (Balf et al. 2018; Rezaie-Balf & Kisi 2018) and water supply (Mounce et al. 2015).

In order to have reliable solutions with comparable results, this paper proposes two data-driven approaches: a CoxMLP-based survival model and an EPR-based model, according to their approved performances in predicting and optimization (as in Awolusi et al. 2018), as well as in modeling failure features (Kvamm et al. 2019), respectively. To enhance the performance of the proposed data-driven approaches, cumulative Weibull distribution (CDF) has been used to develop a statistical reliability model as an additional comparison. The CDF has already been remarkably used in many different applications, such as distribution in life and response time, breakage data, and probabilistic analysis of reliability (Gifty & Bharathi 2020). Since CDF can approximate closely practical distributions, CDF can model numerous failure characteristics while satisfying performances for pipes’ life prognosis and assessment.

This paper mainly contributes:

• -

  To develop several comparable data-driven models to quantify deterioration in pipes using survival, EPR and statistical reliability models. Besides, the correlations between pipes’ failure and different features have also been studied. Through analysis using state-of-the-art methods, diameter, material, and age have been considered as the most influencing features to represent pipes’ failure rate. Pipes’ life prognosis is carried out by the survival model, which consists of an NN with Cox nonlinear proportional hazards regression (CoxMLP). CDF is used to model the failure rate using techniques from reliability theory.

• -

  To predict the RUL of pipes through pipes’ life evolution using the aging models obtained from data.

• -

  To create an easy-reading checklist with risk-level categories and corresponding advice to assist water managers in the daily operation, maintenance, and investment of their water supply assets.

• -

  A real-life demonstration, based on the Barcelona WDN case study, has been used to validate the performance of the proposed approaches.

Following this section, the considered case study is first presented in order to prepare a dataset for use later on. A preliminary analysis of the different features of pipe failures is provided in the ‘Case Study’ section. Afterward, all the proposed modeling approaches, which include the CoxMLP-based survival model, the EPR model, as well as reliability model based on CDF, are presented in the ‘Proposed Approaches’ section. The demonstration and application results of all the proposed approaches based on the Barcelona WDN case study are provided in the ‘Results’ section, including the checklist generated for the water industry to assist operators in planning their maintenance or renewal. Conclusions and future work are discussed in the ‘Conclusions’ section.

The case study used in this paper is based on the Barcelona WDN. The considered database contains information from year 2002 to 2015, where 153,285 event failures have been recorded, including 15 other relevant related variables. In order to prepare a valid dataset for analysis and validation, useful data records were extracted from the raw measurements while focusing only on the pipe failure records with the useful information we need. The records without complete data were deleted.

In general, the use of more feature/factor inputs could potentially lead to a better model but more data measurements are also needed, which also adds difficulties and uncertainties to practical implementation. For developing approaches that are much easier to apply in practice, a preliminary analysis is conducted first to determine the importance of different features. Only the most influencing features will be taken into account for predicting failure rate temporal evolution later on.

Two data-driven approaches, a CoxMLP-based survival model and an EPR model, are developed for data-based pipe prognosis, according to their performances in prediction (Awolusi et al. 2018) and modeling numerous failure characteristics (Kvamm et al. 2019). As an additional comparison for data-driven methods, a CDF-based statistical reliability model has also been presented.

The application of all the approaches follows the same procedure:

Step 1: Failure rate calculation: Based on the historical database, the pipe failure rate (1) is calculated for each tuple (material, age, and diameter).

Step 2: Training/validation phase: A model for the pipe failure rate is built for each tuple obtained in Step 1 using 70% of the data in the considered database. The remaining 30% of the data are used for model validation.

Step 3: Prediction phase: The trained/validated model is used to forecast the evolution of the failure rate in the future (several years ahead), determining the RUL and the maintenance checklist.

According to the NN prediction performance and flexibility while modeling the failure times continuously (see e.g. Awolusi et al. 2018; Kvamm et al. 2019), in this paper, the function is obtained using a multilayer perceptron (NN instead of considering a linear predictor (⁠⁠) as in the classical Cox model (Scheidegger et al. 2015). This is why this method is called CoxMLP The constraint of the proportionality of this model has not been affected by this parameterization process of the failure rate function (2). Moreover, this paper also proposes a parametric method without requiring grouping the matching features. Time can used as a regular co-variate ⁠, which permits interactions between time and other co-variate. Similar to survival analysis, time-dependent co-variate can be taken into account to model the non-proportional effect of a co-variate

The EPR model is an evolutionary computation-based data-driven approach, which can handle pseudo-polynomial structures and represent a physical system precisely. There are several steps in the EPR model: (1) EPR finds suitable model structures using a genetic algorithm based on an evolutionary procedure; (2) carrying out a linear regression using least square optimization in order to compute the model constants. More detailed mathematics about EPR can be found in Berardi et al. (2008) and Mounce et al. (2015), who have already proposed the EPR method to build models for different materials in WDNs. For the pipes without known failures, the same model coefficient as given by EPR is used; for the pipes with failures being monitored, the model coefficient is computed by their historical burst data (Berardi et al. 2008).

In this work, both healthy and faulty pipes are characterized jointly using linear regression to the model coefficient parameter for every material. The pipes that did not have any faults are also considered to carry out the normalization step so that the failure rate over time can be modeled independently from whether the failure exists. This explains why the number of the recorded faults is larger than or equal to 0, ⁠. The method to obtain the model coefficient parameter for different materials is as follows:

• Failures for each age at different years of the monitoring period have been computed and accumulated. The cumulative failure numbers are then normalized considering total pipe numbers, as well as the length of each material.

• Compute the model coefficient parameter for the failures at the same age and at the same monitoring year.

• Compute the model coefficient parameter for each material by weighing each with the number of total failures for each age over the total number of failures.

Once the probability distributions for all pressure zones have been obtained, we will carry out an analysis of the RUL for a certain material. Knowing the total materials in each zone will help us to discover the weight of each material in different zones. This will enable obtaining the joint probability of materials in a particular pressure zone. Considering the failures in different pressure zones, only the zones with the highest number of failures have been selected, for illustrative purposes, from a total of 157 pressure zones:

• -

  Pressure zone 10101 (with 14,048 failures);

• -

  Pressure zone 10004 (with 12,470 failures) and

• -

  Pressure zone 10103 (with 12,197 failures)

The motivation behind this verification is to ensure that failure probabilities obtained per pressure zone considering the material distribution in that pressure zone coincide to a great extent with the failure probabilities obtained per pressure zone. This results in a better understanding of the failure probability of each feature and also allows verifying that the probability computation for each material at a macro-level extrapolated to a specific pressure zone.

According to the explanation and discussion in previous subsections, the CoxMLP can predict the failure probability based on features that have been selected in Section 2. Besides that, CoxMLP can also be applied for pipe failure prognosis. To proceed with the pipe prognosis, we have predicted pipe status using a 25-year ahead horizon with a 5-year step. The features we consider are the most influential ones, the material, the diameter, and also the age. Furthermore, in the considered period, we will also compute the failure rate, using Weibull distributions and the built EPR model.

To prepare for the prediction and prognosis, we have listed material types and the percentage of usage of each material in the Barcelona WDN. The summary is as follows. 18 different types of material have been used in the Barcelona WDN. Among these, ductile iron has been used mostly, with 41.69%; after that, gray cast iron covered 22.35% of the network and asbestos cement corresponded to 13.92%. The rest of the most used materials are used in the following percentages: high-density polyethylene (10.88%), low-density polyethylene (5.60%), reinforced concrete (1.50%), and reinforced concrete welded joint (1.48%). As we cannot collect enough data for the material type with less usage percentage in the prognosis process, we can only focus on the material type and diameter with a higher usage percentage. Therefore, the material types that we are going to implement pipes’ failure probability prediction include asbestos cement, gray cast iron, and ductile iron. In addition to material, diameter, and age will also be considered during the prediction and prognosis processes.

In this section, the proposed approaches are validated using the Barcelona WDN presented in Section 2. The results show the considered case study:

• -

  comparable results among survival CoxMLP models, EPR models, and CDF models in predicting pipe failures in future;

• -

  comparable performance between the survival CoxMLP model and CDF model regarding the RUL over time;

• -

  easy-reading maintenance and renewal plan for the coming 25 years in the form of a checklist.

The Barcelona WDN includes different types of pipes due to their materials, installation periods, diameters, and construction processes. The failure rate is calculated using (1) for all the methods as discussed at the beginning of Section 3. As the calculation is a kind of prediction, we will accumulate the new leakages. As shown in Table 1, the prediction of the pipe failure probability will cover the three main materials and most common diameter (100 mm) in the coming 25 years, with a 5-year time step, using the three proposed methods. While using the EPR approach, we consider the parameter for each material that has been estimated using historical data and the procedure described in the work of Berardi et al. (2008) 

• -

  = 0.0560 for the asbestos cement material;

• -

  = 0.0858 for the gray cast iron material;

• -

  = 0.0199 for the ductile iron material.

In the case of pipes with a current age greater than 60 years, it is highly recommended to carry out an intensive monitoring of the evolution of the pipe because the prediction for all materials and diameter ranges indicates that in less than 5 years it will enter the high-risk zone.

In order to compare clearly the performance of the two approaches, we add more details to Table 2. As this additional comparative analysis has been implemented in a subset of the entire network, the conclusions apply at the subset level with limitations to the entire set. In Table 2, as presented in the column, it is clear that, after considering age and material using CDF (second method), the material with a higher usage percentage has more failures. As the method using CoxMLP (first method) needs more calibrated parameters, the RUL percentage is lower because each intervening factor adds given weight to the total failure. Not all the factors will be considered in the second method, which may decrease the factual percentage. Moreover, as the number of failures decreases for the less used materials, the CoxMLP method shows that the RUL is lower with a higher percentage of failures. However, including more factors using the CoxMLP method will permit us to characterize more precisely pipes behavior.

Table 2

RUL regarding material

RUL regarding material (probability in % of pipe failures) .
Material (presence) .	Date (years) .	.	First method .	Second method .	Difference (%) .
Asbestos cement (13.92%)	(56–101)	1,844	70.8%	74.9%	4.1%
Gray cast iron (22.35%)	(63–101)	4,258	63.8%	67.2%	3.4%
Ductile iron (41.69%)	(59–102)	2,985	67.8%	71.4%	3.6%

RUL regarding material (probability in % of pipe failures) .
Material (presence) .	Date (years) .	.	First method .	Second method .	Difference (%) .
Asbestos cement (13.92%)	(56–101)	1,844	70.8%	74.9%	4.1%
Gray cast iron (22.35%)	(63–101)	4,258	63.8%	67.2%	3.4%
Ductile iron (41.69%)	(59–102)	2,985	67.8%	71.4%	3.6%

View Large

From Table 2 it is important to note how some materials present more problems than others. For example, the gray cast iron has more failures than the ductile iron even though this last one has more presence in the WDN. From this, we can deduce that the gray cast iron is a more problematic material for the pipes than the Ductile Iron.

After presenting all the predictions, prognosis, and analysis results, our final aim is to assist wisely the WDN managers or operators in deciding how to proceed with the renewal plans. Based on the obtained results, an easy-reading checklist has been developed for pipeline monitoring and maintenance plans under diameter, age, and material features. The checklist includes three types of operations: continue to work, should be supervised, and should be replaced according to the status analysis of their pipes in terms of A, B, C, and D as four assessment zones. The procedure of building up this checklist is elaborated as follows:

• (1) Zone A corresponds to the period that does not need assessment, as in zone A, the failure probability is quite low.

• (2) We will start the assessment from the B and C zones, where zone B refers to the [65–70]% of the age while zone C refers to the [75–80]% of the age. Zones B and C represent the failures in relatively low appearance.

• (3) Zone D corresponds to the period where the failure rate indicates that there is a very high probability of failing without needing an assessment.

B and C zones are used as thresholds to evaluate possible operations of the pipes, should they continue to work, need to be supervised, or need to be replaced immediately. The important parameters of age, diameter, and pressures, which have a strong impact on the failure occurrence, will be considered. More explanations about Zone B and Zone C are given as follows:

Zone B:

• -

  Among the three important parameters, there is only one parameter that has values higher than the defined threshold, which will activate the monitoring operation. Thresholds for different features are determined from the obtained results and practical knowledge of the end-users.

• -

  Among the three important parameters, there is more than one parameter that are higher than the thresholds, which will activate the asset replacement operation.

Zone C:

• -

  Among the three important parameters, there is only one parameter that has a value higher than the defined threshold, which will suggest the asset replacement operation.

• -

  Among the three important parameters, there are more than one parameter that are higher than the thresholds, so, as in Zone B, it will activate the asset replacement operation.

A checklist for gray cast iron pipes is extracted for Zone B and Zone C. Moreover, the assessment of a pipe of 59 years (asset enters monitoring phase) is shown in Table 3, where WAR (Within Allowed Range of value) and OAR (Outside Allowed Range of value) mean that a certain parameter is within or outside the allowed range of values, respectively.

Likewise, a checklist has been carried out to incorporate failure prediction in the next 25 years. It provides additional information to the water managers, by means of time intervals in years that are inside of three assessment zones: low, medium, and high risk, following the same criteria applied for zones A, B, and C, that is, less than 65%, [65–70]% and [75–80]% of the asset age, respectively. Thus, a given low-risk zone age interval of [0–10]% shows that the given material will enter the medium-risk zone after 10 years where the risk of asset failure is already significant. The parameter age (years) has been divided into three intervals [0–30], [30–60], and [60 – ]. Regarding the materials, the asbestos cement types, the gray cast iron types, and the ductile iron types have been included. The diameter (mm) has been split into three intervals [0–100], [100–150], and [150 – ]. This will facilitate the adoption of maintenance tasks for certain properties of a pipe. A checklist has been given in Table 4.

This paper has proposed multiple comparable data-driven approaches for pipe life prognosis in water distribution networks from both scientific and practical perspectives. Quantitative analysis has been carried out for the first time to evaluate the impact of different influential factors (mainly material, diameter, and age) on the pipe's failure . Occurrence regulations of pipe failures are revealed in a comparable way through the CDF-based statistical reliability model, the CoxMLP-based survival model, as well as EPR, an EPR model. Similar performance in predicting future failure rates confirms the consistency of the proposed approaches. The RUL evolution in around 120 years with no more than a 5% difference between the survival and reliability models has deepened these similarities. The survival model, based on CoxMLP, works in the considered case study with lower RUL because there are more factors involved. Furthermore, the conclusion can also be made that more failures happen to the materials with higher usage percentages in the network. To apply wisely this research into practice, an easy-reading checklist has also been provided, which divides all pipes into A, B, C, and D as four assessment zones with corresponding operations continue to work, should be supervised, and should be replaced. To enhance clarification, a failure probability for the future 25 years is also predicted and added in an easy-reading table using low, medium, and high to evaluate risk levels for each pipe. This work contributes to asset monitoring and maintenance from both scientific and practical perspectives. In future research, the information extracted for this study will be used to enhance the current leak localization methodologies based on pressure monitoring and also to develop optimal pipe renewal plans, trying to obtain the best tradeoff between investment and quality of service.

Data cannot be made publicly available; readers should contact the corresponding author for details.

The authors declare there is no conflict.